[pulseaudio-discuss] Musings on stream device and volume restore rules

[Lennart Poettering]

On Tue, 24.03.09 21:08, Colin Guthrie (gmane at colin.guthr.ie) wrote:

Sorry for the long delay, but here are a few notes from my side:

>
> Stream Memory Database and Restoration Rules
> ============================================
>
> This document outlines the needs and use case for the stream memory  
> capabilities in pulseaudio and how this information should be used. It  
> finishes with recommendations for implementation.
>
> Restrictions
> ============
>
> The system employed needs to be simple and allow for fully server based  
> management and application of the rules. The logic should be kept simple  
> with minimal overhead but still be powerful enough to meet all the needs  
> of current actual and potential use cases.
>
> Current System
> ==============
>
> Currently module-stream-restore stores various metadata on streams it  
> sees. This currently consists of:
>  * volume
>  * channel map

The channel map is only stored to make sense of the volume when
restoring. It is not data that would be restored or could be
restored. Channel maps is something applications have to decide about,
not m-s-r.

>  * mute status
>  * device (optional)
>
> It stores this metadata under a key that governs how it is matched again  
> in the future.
>
> The key itself has two parts, a prefix from a predefined list and a  
> suffix relating to the specific stream in some way.
>
> Prefixes include:
>  * sink-input-by-media-role
>  * sink-input-by-application-id
>  * sink-input-by-application-name
>  * sink-input-by-media-name
>
> (where "sink-input" could also be "source-output" for input streams)
>
>
> The specific identifier is suffixed on the end of this, to create a  
> final key that is stored and queried in the database to give final keys  
> of e.g.:
>  * sink-input-by-media-role:event
>  * sink-input-by application-name:MPlayer
>  * etc.
>
> When a stream is started, module-stream-restore will pick exactly one  
> key that it will use to identify a stream. If the stream has a media  
> role property it will use *-by-media-role. It then falls back to check  
> the different identifiers, in order, as outlined above.

Two more notes:

PA will only save volume/mute/device for streams where an explicit
user-triggered volume change/mute change/device change was done. It
won't create/changes rules for volume/mute/device changes that were
done due to some automatic internal action, such as hot-unplug, ...

Also, if no rule is existant the stream volume is set to 100% relative
to the current sink volume, the mute status is set to UNMUTED and the
device is set to the default device that has been configured for the
server or the first available sink (for whatever that means) of the server.

> Problems
> ========
>
> The current system has a couple limitations.
>  1. If a stream has a media role, it is impossible to save a volume for  
> the specific application with that role as the key  
> sink-input-by-media-role:xxxx will always take precidence over a  
> sink-input-by-application-name:yyyy keyed rule.

No sure if that is a limitation or a feature. 

>  2. If an application is modified to start providing new metadata (e.g.  
> a role) the entries for that specific application that were used  
> (sink-input-by-application-name) will no longer be matched and the data  
> will be stale in the file. This is not really a major problem.
>  3. The current system only allows for one device to be matched, not an  
> ordered priority list.

Other limitations are:

It is a bit unclear whether it even makes sense to restore the volume
on a different device than it was initially configured on. i.e. 100%
on my hifi deck might be something completely different than 100% on
my headset. To remedy this we only restore the volume for rules where
we also store the device. The result is that you can have rules for
matching devices, and for matching devices+volumes, but not volumes
alone. It's a bit weird. Sometimes you might not want to restore the
device but still restore the volume if its the same device you end up
using.

There is no sane way to say 'move *all* streams and rules to this new
device'. Same for 'set *all* streams volume to x%".

We don't store any information about devices that are currently not
available, except sometimes the name name as saved by m-d-r. But that
we don't really have either. Icons, device descriptions, ... are never
stored. 

Also, there is no logic to move a stream to a newly plugged in device
automatically. But there's some simple logic to move a stream away
from a device that is being unplugged. Having this is highly
desirable.

Also, sometimes a newly created audio streams might result in change
of configuration. i.e. when a phone call stream is created it makes
sense to switch a bt headset from a2dp to hsp mode, i.e. causing one
sink to be replaced and an additional source to become available. That
means we probably shouldn't be binding our rules to sinks/sources but
to cards. Also, rules might trigger card profile changes.

Then, some rules might be good to have as 'implicit'
defaults. i.e. when the user didn't configure anything otherwise
streams marked as 'phone call' should probably go to a device that is
marked as 'headset' -- and not to the spdif output of the internal card.

Finally, with the arrival of jack sensing things get even more
complicated. i.e. music should go out via spdif when that is
plugged in, but the bt headset otherwise with a last fallback of the
internal speakers. Now, suddenly it's not enough to save/restore
information about which device to pick, but about to which card
to pick given a specific profile is set, or even wich card to pick and
which profile to change to.

> NB 3 is only strictly problemati in the case of Phonon integration which  
> uses such a scheme.
>
> Use Case
> ========
>
> To think about what we need to change in the system we need to look at  
> what the user wants out of the system.
>
> For the most part we want routing rules to be automatic, but still allow  
> the user the degree of control they need.
>
> For comparison, Phonon has a device priority list system that allows  
> each role to prioritise the output devices they want to use. When a  
> stream with that role is played, it will go through the list of devices  
> until it finds one that is currently avaialble (i.e. it keeps track of  
> devices that are no longer present - e.g. USB sound cards) and uses  
> that. It offers this priority list for each role, as well as a "global"  
> list, which is used when no role is provided by the application.  
> Generally speaking, users seem happy with this arrangement and this  
> degree of flexibility (albeit there are not many Phonon applications to  
> expliot this flexibility fully yet.

KDE users may be happy about that 'flexibility', but they are the ones
who also think it is a good idea to ask for MySQL credentials when you
start their media player. ;-) 

People have requested a lot of things from PA, but not many have
requested having such a publicly configurable priority list...

Let's face it, 80% of all folks only have one audio device, the one
built into their computer. And then there are a maybe 19% who might
have one additional BT or USB headset/headphones or one additional USB
sound card. And then there are 1% who have more than two audio
devices. And for one or two sound cards having a priority list doesn't
really make sense, does it? Also, don't forget that using multiple
devices really *simultaneously* is very seldom.

Now, I am not saying that having such a priority list of devices is a
bad idea. But I think the use case of folks with gazillions of devices
is not realistic.

(I) A better use case: the user plugs in his new audio device and now
wants to play his music via that device. He wants to tell the computer
that once in a simple way and wants that his music just moves to that
device instantly and have the computer remember for the next time and
that's it. He doesn't want a full screen dialog of lot's of options he
doesn't understand and most importantly he doesn't want to understand
the concept of 'priority lists')

(II) Or another use case: at work the user wants his music to be
played on his BT headset. While travelling the user wants his music to
be played on the internal speakers. While at home he wants his music
to be played on his expensive USB sound card which connects to his
stereo. For him it doesn't make sense to priorize the bt headset
vs. the usb sound card, because he isn't at work and at home at the
same time.

(III) A third use case: the user wants music and events sounds and
everything else go through his internal speakers. Phone calls however
should go through his BT headset if available. 

(IV) A fourth use case: some app played a very short sound. The user
wants to adjust the sound's volume and device although it is already
finished, for future streams.

> Looking at native pulse applications, the "default" sink option in  
> pavucontrol (or more correctly in the daemon itself) is the most  
> commonly misunderstood option and amounts to by far and away the most  
> common support issue: "I've changed my default to my USB but application  
> X still plays sound via my Interal card". From an internal prespective,  
> we do not consider this to be a "default" sink at all, but rather a  
> fallback and arguably calling it "default" was a mistake, but the fact  
> remains that users want an "active" default capability in the system  
> somewhere, where changing the default device will actually move any  
> active streams across to that device if they are currently using a  
> different default. Probably the second most common support question is:  
> "I set my USB to default, but then unplugged it and the application  
> started playing on my internal speakers... Great! But when I plugged my  
> USB back in it didn't move it back. What's up?".
>
> What to do
> ==========
>
> With the above use cases in mind, it's clear that the current system  
> needs to be revised to cope. In summary the problems we want to address 
> are:
>  1. Allow for an "active" default

Smells like an UI issue to me mostly.

>  2. Allow for a prioritised list of devices to be created (on a per-role 
> basis - finer grained control would be too much).

We probably indeed want this. However I doubt that this should be user
visible, or even controllable from outside of the PA server. It should
be kept internally and learned internally. I don't think it makes
sense to show the user devices that are not plugged in. A simple
solution could be to use a list where the currently selected device if
it was explicitly selected is moved to the top. i.e. just a simple
history. And we pick the most recent item from the history list that
would currently work.

>  3. Allow for applications which have the same role to have independant  
> volumes saved.

Sure? Use case?

> To this end, I think the current volume/map/mute/device database is  
> incorrect. With the flat volume support, I think per-application volumes  
> are more appropriate, rather than per-role.

Really? What's the difference between playing back music in rhythmbox
and in banshee? The volume should be exactly the same. And what's the
difference between getting an event sound from firefox or from
nautilus? Or getting a phone stream from ekiga or telepathy?

I think storing things per-application is only a good idea if they
don't supply us with the role they belong to or because they don't
fit in any of the existing roles.

> I think that a key->device->priority table is needed to allow for a  
> given role identifier to have it's own rules written appropriately.  

> Pulse needs to remember the sinks (and sources) both past and present  
> (including their "nice" name (description)) and provide a way for a UI  
> app to purge a currently unavailable sink (e.g. remove it from the 
> cache).

Again, I don't think it is a good idea to clobber things with audio
devices that the user might have possessed at one time. For each
role/active stream the user just should need to select one as the one
to use from a list of actually available choices. If later on a
different selection of devices is available it's PA's job to come up a
with a good replacement choice by looking at the history of the user's
choices.

> When a stream is encountered, it's role should be checked and the  
> prioritised list of devices loaded for that role. As in Phonon, if a  
> stream does not have a role a "default" priority list should be used.  
> (Optional) A subsequent check should be made to see if any specific  
> rules state that this individual application has a preferred device to  
> allow for fine grained control. If such a rule exists the device should  
> be move to the top of the already loaded priority list or injected there  
> if it does not exist (unlikely). The stream is then restored to the  
> first available device. This priority list should be saved with the  
> stream so it can be easily re-evaluated at any point should new sinks  
> become available later.
>
> If any priority list is changed (e.g. by a client application) all  
> streams should have their cached priority list invalidated and rebuilt  
> (perhaps the rebuilding can be lazy - e.g. it's only loaded when
> needed?)

I don't follow.

> With regards to volumes etc., these should be saved against a given  
> application and not a role. To do so at the role level is unlikely to be  
> a useful end user feature.

Again, I doubt that. 

> With this system in place, a neat and effective UI can be produced and a  
> good mapping into KDE can also exist which will be good for cross  
> desktop accpetance of PulseAudio in the linux world.

Hmm, I honestly believe the KDE UI is too complex. Exposing this
priority list to the user IMHO is overkill and we'd never implement
something like that on GNOME.

Let's not limit ourselves by simply looking how KDE, or GNOME, or
Windows, or MacOS do it right now. Let's do the design from
scratch. It is not convincing to me to adopt a particular scheme just
because KDE does it. And that's not because I hate KDE (I don't) but
because I want to design a system that is as good and reasonable as I
can come up with. And to make this clear: I don't look on how GNOME
handles this right now either. 

Let me try to summarize what I have in mind:

Our rules have the following 'left-hand side', i.e. what is being matched on the streams:

A.1) the stream role, in absence we synthesize one by using the program name/identifier
A.2) the stream role + already on a specific device
A.3) the stream role + already on a specific card
A.4) all streams

Additional conditions when rules may or may not apply:

B.1) when a stream is first created
B.2) when a sink/source becomes available
B.3) when a card becomes available
B.4) when the user reconfigures things

We also need to match devices:

C.1) match device
C.2) match card
C.3) match device/card form factor
C.4) match currently selected profile
C.5) all devices

And on the 'right-hand side' (the effect of the rule):

D.1) move to a sink/source, possibly based on 'take first available from list' (aka history list)
D.2) move to a sink/source of a specific card, possibly based on history list
D.3) move to a sink/source of a specific card with history, and switch profile (profile switching with history list, too?)
D.4) set volume
D.5) set mute

(Did I miss anything?)

These are probably the most relevant features of a rule system for the
desktop.  Our particular focus here is configurable rules,
right? Hence I'd suggest ignoring for now static rules like "move
phone calls to devices with form factor 'headsets'". That allows us to
ignore C.3 and D.3 (I think...)

So the right hand side could be encoded in something like this:

struct device_history_item {
       char *device;
       char *card;       
       pa_cvolume volume;  /* Volumes only make sense in the context of a specific device */
       pa_channel_map map; /* To make sense of the volume */
};
struct right_hand_side {
       pa_bool_t set_device_or_card;
       pa_bool_t set_volume;
       pa_bool_t set_mute;
       pa_bool_t muted;
       struct device_history_item *device_history;
};

And the left hand side like this:

struct left_hand_size {
       char *role;
       char *device; 
       char *card;
       char *profile;

       pa_bool_t match_role;
       pa_bool_t match_device_or_card;
       pa_bool_t match_profile;

       pa_bool_t on_stream_new;
       pa_bool_t on_device_or_card_new;
};       

And when we have this most of the 'automatic' rules could be covered
by this. Any use cases that are not?

I probably should explain here how the rules would be automatically
created, and what part would be configurable, but it's already too
late here, so I'll leave that for your immagination... 

Lennart

-- 
Lennart Poettering                        Red Hat, Inc.
lennart [at] poettering [dot] net         ICQ# 11060553
http://0pointer.net/lennart/           GnuPG 0x1A015CC4