[pulseaudio-discuss] Musings on stream device and volume restore rules
Colin Guthrie
gmane at colin.guthr.ie
Sun Apr 12 08:23:42 PDT 2009
'Twas brillig, and Lennart Poettering at 11/04/09 03:25 did gyre and gimble:
>> Currently module-stream-restore stores various metadata on streams it
>> sees. This currently consists of:
>> * volume
>> * channel map
>
> The channel map is only stored to make sense of the volume when
> restoring. It is not data that would be restored or could be
> restored. Channel maps is something applications have to decide about,
> not m-s-r.
OK, so these two can be collectively considered as "volume".
> Two more notes:
>
> PA will only save volume/mute/device for streams where an explicit
> user-triggered volume change/mute change/device change was done. It
> won't create/changes rules for volume/mute/device changes that were
> done due to some automatic internal action, such as hot-unplug, ...
>
> Also, if no rule is existant the stream volume is set to 100% relative
> to the current sink volume, the mute status is set to UNMUTED and the
> device is set to the default device that has been configured for the
> server or the first available sink (for whatever that means) of the server.
Noted.
>> Problems
>> ========
>>
>> The current system has a couple limitations.
>> 1. If a stream has a media role, it is impossible to save a volume for
>> the specific application with that role as the key
>> sink-input-by-media-role:xxxx will always take precidence over a
>> sink-input-by-application-name:yyyy keyed rule.
>
> No sure if that is a limitation or a feature.
Indeed. Personally, it smells like a limitation. I'm not sure I'd want
to my two voip apps to have the same volume for example (they may do
things internally with regards to levels etc, that mean I want to do
things differently.
>> 2. If an application is modified to start providing new metadata (e.g.
>> a role) the entries for that specific application that were used
>> (sink-input-by-application-name) will no longer be matched and the data
>> will be stale in the file. This is not really a major problem.
>> 3. The current system only allows for one device to be matched, not an
>> ordered priority list.
>
> Other limitations are:
>
> It is a bit unclear whether it even makes sense to restore the volume
> on a different device than it was initially configured on. i.e. 100%
> on my hifi deck might be something completely different than 100% on
> my headset.
Very true.
> To remedy this we only restore the volume for rules where
> we also store the device. The result is that you can have rules for
> matching devices, and for matching devices+volumes, but not volumes
> alone. It's a bit weird. Sometimes you might not want to restore the
> device but still restore the volume if its the same device you end up
> using.
So would an application/device -> volume kinda table be wise here? It
would store the volume of the app on that device? This isn't used for
picking which device to restore the app to, that would be stored separately.
This has the arguably strange result that moving a stream to a different
sink may change the volume of that stream which the user may not
expect... e.g play some music quietly on USB and then pull the plug out
and have it blare out noisily from the internal system as the volume
jumps to max... (as that is what was stored previously).
> There is no sane way to say 'move *all* streams and rules to this new
> device'. Same for 'set *all* streams volume to x%".
I think there is a sane way to say 'move all streams to a new device'
tho'. I don't think we should move the rules. If we store a app+device
keyed volume table as mentioned above, we could use that info when we
happen to restore app to device. e.g. that particular rule doesn't need
to be moved.
The only rule we do need to change is the one that says "this app wants
this device".
> We don't store any information about devices that are currently not
> available, except sometimes the name name as saved by m-d-r. But that
> we don't really have either. Icons, device descriptions, ... are never
> stored.
This could be changed tho' (and I think we do need this to create the
kind of UI people will find useful).
> Also, there is no logic to move a stream to a newly plugged in device
> automatically. But there's some simple logic to move a stream away
> from a device that is being unplugged. Having this is highly
> desirable.
Again, I think this is fairly easy to implement... m-s-r (or another
module) just needs to hook into the right events and re-process it's
routing rules when a hotplug event occurs. Ultimately I think that this
functionality will merge in the functionality of m-rescue-streams too.
> Also, sometimes a newly created audio streams might result in change
> of configuration. i.e. when a phone call stream is created it makes
> sense to switch a bt headset from a2dp to hsp mode, i.e. causing one
> sink to be replaced and an additional source to become available. That
> means we probably shouldn't be binding our rules to sinks/sources but
> to cards. Also, rules might trigger card profile changes.
Again, if the routing rules are re-run when a hotplug event occurs this
shouldn't be a problem... I guess there could be circular loops created
if rules trigger profile changes which causes a new sink which re-runs
the rules which triggers a profile change which ..... etc. But I'm sure
this can be protected against in some way.
> Then, some rules might be good to have as 'implicit'
> defaults. i.e. when the user didn't configure anything otherwise
> streams marked as 'phone call' should probably go to a device that is
> marked as 'headset' -- and not to the spdif output of the internal card.
Yeah that makes sense... I think this we have to show this somehow in
the GUI tho' so users can see transparently what is going on.
> Finally, with the arrival of jack sensing things get even more
> complicated. i.e. music should go out via spdif when that is
> plugged in, but the bt headset otherwise with a last fallback of the
> internal speakers. Now, suddenly it's not enough to save/restore
> information about which device to pick, but about to which card
> to pick given a specific profile is set, or even wich card to pick and
> which profile to change to.
/me pretends not to notice this complication... ;)
> KDE users may be happy about that 'flexibility', but they are the ones
> who also think it is a good idea to ask for MySQL credentials when you
> start their media player. ;-)
Not sure what you mean about that as I've never been asked for MySQL
credentials using any KDE media player..., I can't really see it's
relevant here either.
> People have requested a lot of things from PA, but not many have
> requested having such a publicly configurable priority list...
Well perhaps a priority list is not that important, but certainly some
way to configure things with an active default nicely which restores
itself and the streams etc. With this in mind a priority list in terms
of UI presentation is a good solution to this implementation wise.
From the feedback I've received, the actual layout and presentation of
the KDE configuration UI is pretty good.
> Let's face it, 80% of all folks only have one audio device, the one
> built into their computer. And then there are a maybe 19% who might
> have one additional BT or USB headset/headphones or one additional USB
> sound card. And then there are 1% who have more than two audio
> devices. And for one or two sound cards having a priority list doesn't
> really make sense, does it? Also, don't forget that using multiple
> devices really *simultaneously* is very seldom.
True, but again, a list is quite a simple concept. It works IMO pretty
well on OSX as a list too.
> Now, I am not saying that having such a priority list of devices is a
> bad idea. But I think the use case of folks with gazillions of devices
> is not realistic.
True, but I think with BT + USB + Built in + Network and you've already
got a few different sinks there without trying too hard. While I agree
it's going to be a minority of people that have such a setup, I think
making configuration simple and easy in this scenario should be possible
while maintaining a non-confusing UI for the majority, simpler case.
> (I) A better use case: the user plugs in his new audio device and now
> wants to play his music via that device. He wants to tell the computer
> that once in a simple way and wants that his music just moves to that
> device instantly and have the computer remember for the next time and
> that's it. He doesn't want a full screen dialog of lot's of options he
> doesn't understand and most importantly he doesn't want to understand
> the concept of 'priority lists')
Well, I don't think a priority list is that complex and like I say the
feedback from people using the KDE priority list is pretty good. Ditto
with the CoreAudio list of devices.
> (II) Or another use case: at work the user wants his music to be
> played on his BT headset. While travelling the user wants his music to
> be played on the internal speakers. While at home he wants his music
> to be played on his expensive USB sound card which connects to his
> stereo. For him it doesn't make sense to priorize the bt headset
> vs. the usb sound card, because he isn't at work and at home at the
> same time.
Sure, but dealing with that kind of scenario in a UI without it being
very complex will be tricky. With a list, concepts are simple and the
user will happily know that although he's prioritising one over the
other that it doesn't really matter as he simply wont have a setup where
the sinks are plugged at the same time - the logic of what will happen
is still obvious to the user.
> (III) A third use case: the user wants music and events sounds and
> everything else go through his internal speakers. Phone calls however
> should go through his BT headset if available.
Should be OK with a priority list I should think.
> (IV) A fourth use case: some app played a very short sound. The user
> wants to adjust the sound's volume and device although it is already
> finished, for future streams.
Indeed. I think this should just be a matter of exposing the stream
restore db in some capacity. Don't think this need will massively affect
things here...
>> What to do
>> ==========
>>
>> With the above use cases in mind, it's clear that the current system
>> needs to be revised to cope. In summary the problems we want to address
>> are:
>> 1. Allow for an "active" default
>
> Smells like an UI issue to me mostly.
>
>> 2. Allow for a prioritised list of devices to be created (on a per-role
>> basis - finer grained control would be too much).
>
> We probably indeed want this. However I doubt that this should be user
> visible, or even controllable from outside of the PA server. It should
> be kept internally and learned internally. I don't think it makes
> sense to show the user devices that are not plugged in. A simple
> solution could be to use a list where the currently selected device if
> it was explicitly selected is moved to the top. i.e. just a simple
> history. And we pick the most recent item from the history list that
> would currently work.
I'd very much like to be able to query the server for devices that are
not plugged in and show them to the user.
IMO this makes sense anyway (I kinda disagree that we should hide this
completely). I really want this for KDE integration to work nicely.
Their UI *does* expose this level of detail to the user (at a per-role
level, not a per-app) and to get really good integration with PA
(something I really want to see), it would be essential for me to be
able to do this.
Also while the default tools etc. may present the user only simple
options, I don't think we should discount the possibility of a more
complex configuration app for power users.
That said, I do like the "history" idea. Perhaps the management of this
history can be modular, so that in a KDE integrated mode, we leave it to
the UI they provide to configure the lists, but for a "standard"
install, the priority list is managed automatically by this new module?
Would that work?
>> 3. Allow for applications which have the same role to have independant
>> volumes saved.
>
> Sure? Use case?
Two VoIP apps that have completely different levels coming out.
>> To this end, I think the current volume/map/mute/device database is
>> incorrect. With the flat volume support, I think per-application volumes
>> are more appropriate, rather than per-role.
>
> Really? What's the difference between playing back music in rhythmbox
> and in banshee? The volume should be exactly the same. And what's the
> difference between getting an event sound from firefox or from
> nautilus? Or getting a phone stream from ekiga or telepathy?
Well rhythmbox may implement AGC and banshee may not. Perhaps one
implements it's on soft volume and the other doesn't.
It's maybe not a massive problem practically tho', so I'm not sure if
this is imporant really.
> I think storing things per-application is only a good idea if they
> don't supply us with the role they belong to or because they don't
> fit in any of the existing roles.
Perhaps. I'm still a bit undecided here personally. If this is the case,
I'd argue that per-app volume control should be completely removed, and
all "uncategorised" apps (e.g. those sans-role) be lumped into a
"Uncategorised" role. It is this role that then has it's volume
adjusted, not the app. This is clearly much simpler, but it also reduces
the flexibility of pulseaudio by a big chunk. But I think if you're
going to hide features under a certain condition (stream sans role),
that it should be consistent across the board.
>> I think that a key->device->priority table is needed to allow for a
>> given role identifier to have it's own rules written appropriately.
>
>> Pulse needs to remember the sinks (and sources) both past and present
>> (including their "nice" name (description)) and provide a way for a UI
>> app to purge a currently unavailable sink (e.g. remove it from the
>> cache).
>
> Again, I don't think it is a good idea to clobber things with audio
> devices that the user might have possessed at one time. For each
> role/active stream the user just should need to select one as the one
> to use from a list of actually available choices. If later on a
> different selection of devices is available it's PA's job to come up a
> with a good replacement choice by looking at the history of the user's
> choices.
I agree on the most part, but again, if this stuff is more or less
stored internally, would it be a terrible idea to create a module
capable of doing this. To integrate nicely into KDE I'd really need this
functionality. I think it can be done in a modular way tho', so
shouldn't get in the way of a default setup. WDYT?
>> When a stream is encountered, it's role should be checked and the
>> prioritised list of devices loaded for that role. As in Phonon, if a
>> stream does not have a role a "default" priority list should be used.
>> (Optional) A subsequent check should be made to see if any specific
>> rules state that this individual application has a preferred device to
>> allow for fine grained control. If such a rule exists the device should
>> be move to the top of the already loaded priority list or injected there
>> if it does not exist (unlikely). The stream is then restored to the
>> first available device. This priority list should be saved with the
>> stream so it can be easily re-evaluated at any point should new sinks
>> become available later.
>>
>> If any priority list is changed (e.g. by a client application) all
>> streams should have their cached priority list invalidated and rebuilt
>> (perhaps the rebuilding can be lazy - e.g. it's only loaded when
>> needed?)
>
> I don't follow.
That's a pity as I still think this is right way forward.
Pseudo Logic:
1. New stream is created.
2. role = (empty(role) ? "default" : role);
3. Load role priority list and store with stream.
4. Check for per-app device preference, if exists, manipulate priority
list accordingly.
5. Find first available device in priority list and use it.
(NB step 4 should be modular - e.g. modules should be able to hook into
this in some way to manipulate the priority list as they need - this
should give sufficient flexibility).
Event: New USB sink plugged in.
1. Go through all streams.
2. Find first available sink in the stream's priority list.
3. If the results of 2 != current, move the sink input.
(this allows the scenario that a user has chosen that their music should
come out of USB speakers, they can plug/unplug their USB speakers as
much as they want, and their currently playing track will automatically
jump to the USB when it's there and fall back to some other device when
it's not! Perfect!).
Event: USB sink pulled out.
* Do exactly the same as "New USB sink plugged in"
This replaces the functionality of module-rescue-streams.
Event: User changes a "priority list" via some UI.
1. Go through all streams.
2. Invalidate & Reload priority list.
3. Re-evaluate priority list (i.e. do the same as "New USB sink plugged in")
>> With regards to volumes etc., these should be saved against a given
>> application and not a role. To do so at the role level is unlikely to be
>> a useful end user feature.
>
> Again, I doubt that.
Perhaps. I'm not really decided on this one as I mentioned above.
>
>> With this system in place, a neat and effective UI can be produced and a
>> good mapping into KDE can also exist which will be good for cross
>> desktop accpetance of PulseAudio in the linux world.
>
> Hmm, I honestly believe the KDE UI is too complex. Exposing this
> priority list to the user IMHO is overkill and we'd never implement
> something like that on GNOME.
That as it may be, but I'd very much like to integrate pulse into KDE
properly, as do many others. Their UI is their UI, whether you or I like
it is not really up for debate here. The feedback from users I've read
is very good. People know how this system works and can do what they
want with it, so it must be working in some capacity.
> Let's not limit ourselves by simply looking how KDE, or GNOME, or
> Windows, or MacOS do it right now. Let's do the design from
> scratch. It is not convincing to me to adopt a particular scheme just
> because KDE does it. And that's not because I hate KDE (I don't) but
> because I want to design a system that is as good and reasonable as I
> can come up with. And to make this clear: I don't look on how GNOME
> handles this right now either.
That's fine, but just like you had to create an also plugin to make
pulse work with what's out there, we may have to do something similar to
make pulse work with various higher level stuff.
In KDE just now for example, I only display one thing in the "device"
list, and that is a big PulseAudio "device". It would be nice to be able
to see a list of sinks (past and present) here instead. This is the
framework I need to fit into, whether it is the pinnacle of UI design or
not. So while I agree with your "design from scratch" in principle, we
do need to remain slightly flexible here.
>
> Let me try to summarize what I have in mind:
>
> Our rules have the following 'left-hand side', i.e. what is being matched on the streams:
>
> A.1) the stream role, in absence we synthesize one by using the program name/identifier
> A.2) the stream role + already on a specific device
> A.3) the stream role + already on a specific card
> A.4) all streams
>
> Additional conditions when rules may or may not apply:
>
> B.1) when a stream is first created
> B.2) when a sink/source becomes available
> B.3) when a card becomes available
> B.4) when the user reconfigures things
>
> We also need to match devices:
>
> C.1) match device
> C.2) match card
> C.3) match device/card form factor
> C.4) match currently selected profile
> C.5) all devices
>
> And on the 'right-hand side' (the effect of the rule):
>
> D.1) move to a sink/source, possibly based on 'take first available from list' (aka history list)
> D.2) move to a sink/source of a specific card, possibly based on history list
> D.3) move to a sink/source of a specific card with history, and switch profile (profile switching with history list, too?)
> D.4) set volume
> D.5) set mute
>
> (Did I miss anything?)
>
> These are probably the most relevant features of a rule system for the
> desktop. Our particular focus here is configurable rules,
> right? Hence I'd suggest ignoring for now static rules like "move
> phone calls to devices with form factor 'headsets'". That allows us to
> ignore C.3 and D.3 (I think...)
>
> So the right hand side could be encoded in something like this:
>
> struct device_history_item {
> char *device;
> char *card;
> pa_cvolume volume; /* Volumes only make sense in the context of a specific device */
> pa_channel_map map; /* To make sense of the volume */
> };
> struct right_hand_side {
> pa_bool_t set_device_or_card;
> pa_bool_t set_volume;
> pa_bool_t set_mute;
> pa_bool_t muted;
> struct device_history_item *device_history;
> };
>
> And the left hand side like this:
>
> struct left_hand_size {
> char *role;
> char *device;
> char *card;
> char *profile;
>
> pa_bool_t match_role;
> pa_bool_t match_device_or_card;
> pa_bool_t match_profile;
>
> pa_bool_t on_stream_new;
> pa_bool_t on_device_or_card_new;
> };
>
> And when we have this most of the 'automatic' rules could be covered
> by this. Any use cases that are not?
Well, at a higher level than I'd like to see the implementation, I think
this is OK.
As I said above, I'd still prefer to see pulse maintain a priority list
internally, and have a module-history "manage" that list automatically
to achieve the result you want. That way the routing logic can follow
the process I outlined above ("Pseudo Logic").
I think the restoration of volume is slightly different tho' and could
possibly be handled totally outwith the "priority list" approach (which
is all about device restoration). I'm not really sure tho'.
We're possibly thinking about the problem differently and thus have
different ways of visualising it, which probably isn't helping.
Hopefully I've managed to get my ideas across a bit more clearly this time.
Col
--
Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/
Day Job:
Tribalogic Limited [http://www.tribalogic.net/]
Open Source:
Mandriva Linux Contributor [http://www.mandriva.com/]
PulseAudio Hacker [http://www.pulseaudio.org/]
Trac Hacker [http://trac.edgewall.org/]
More information about the pulseaudio-discuss
mailing list