[pulseaudio-discuss] Bluetooth codec selection

[Pali Rohár]

On Tuesday 17 March 2020 11:23:22 Tanu Kaskinen wrote:
> On Sun, 2020-01-12 at 12:42 +0100, Pali Rohár wrote:
> > On Sunday 12 January 2020 09:21:31 Tanu Kaskinen wrote:
> > > On Tue, 2020-01-07 at 19:37 +0100, Pali Rohár wrote:
> > > > Hello!
> > > > 
> > > > My A2DP patch series which adds support for more A2DP codecs is still in
> > > > review state, but Tanu about year ago wrote that do not like my proposal
> > > > with usage of PA profiles for selecting A2DP codec. Because nobody for
> > > > last year come up with another option how to solve this problem and
> > > > because without it is not possible to add support for other codecs it
> > > > means that pulseaudio is stucked and has no way to improve A2DP support.
> > > > 
> > > > So here I'm adding another proposal how to handle multi-codec support
> > > > for bluetooth in pulseaudio. But first description of our problem:
> > > > 
> > > > Currently we have following profiles for bluetooth card:
> > > > 
> > > > * headset_audio_gateway
> > > > * headset_head_unit
> > > > * a2dp_source
> > > > * a2dp_sink
> > > > 
> > > > If we look at HW level of HSP/HFP profiles they may supports following
> > > > codecs: CVSD, mSBC, AuriStream. All are bidirectional, synchronous.
> > > > Moreover encoding from PCM stream to codec data may be done at HW level
> > > > (bluetooth adapter) or on SW level (pulseaudio). Currently encoding to
> > > > CVSD is done at HW level and encoding to mSBC at SW level. But e.g. new
> > > > Thinkpads has bluetooth adapters which can do mSBC-encoding at HW level
> > > > and theoretically pulseaudio could just pass PCM samples to BT without
> > > > need to use any software sbc encoding library (*).
> > > > 
> > > > And if we look at A2DP profiles it is quite a bit complicated. Every
> > > > "A2DP codec" is either one-way (source or sink) or is bi-directional
> > > > (but it is rare and most devices does not support them). Plus every
> > > > "A2DP codec" may support more "codec profiles", so e.g. we have SBC-MQ
> > > > or SBC-HQ. And bidirectional "A2DP codecs" may use different "audio
> > > > codec" for one directional and different for back directional (e.g.
> > > > A2DP codec aptX-LL uses aptX codec for playing and SBC codec for
> > > > recording).
> > > > 
> > > > Result is: "SBC" codec is supported in "headset_head_unit" PA profile
> > > > (as mSBC"), it is supported also in "a2dp_sink" PA profile when "A2DP
> > > > SBC" codec is used and also in "a2dp_source" PA profile when "aptX-LL"
> > > > codec is used.
> > > > 
> > > > So adding a new API which sets codec on PA bluetooth card is not enough.
> > > > 
> > > > I would propose following API:
> > > > 
> > > > Add support for pulseaudio "sub-profiles". For every pulseaudio profile
> > > > it would be possible to register sub-profile and applications via
> > > > pulseaudio API would be able to change sub-profile of some PA card (if
> > > > currently selected PA profile would support it).
> > > > 
> > > > Plus adds a two new profiles:
> > > > * a2dp_source_with_backchannel
> > > > * a2dp_sink_with_backchannel
> > > > 
> > > > For music playback in most cases is useful only profiles without
> > > > backchannel to maximize bandwidth. And because A2DP codecs with
> > > > bachannel does not fit into HSP profiles it make sense to have them in
> > > > separate profile. In most cases A2DP profile with backchannel is the
> > > > best way for VOIP (as it has microphone recording support), so bluetooth
> > > > policy plugin would be extended to use also this profile for VOIP.
> > > > 
> > > > Sub-profiles for bluetooth PA card would be following:
> > > > 
> > > > * headset_audio_gateway
> > > >   - CVSD (hardware)
> > > >   - mSBC (software)
> > > >   - mSBC (hardware)
> > > >   ...
> > > > * headset_head_unit
> > > >   - same as in headset_audio_gateway
> > > > * a2dp_source
> > > >   - SBC-LQ
> > > >   - SBC-MQ
> > > >   - SBC-HQ
> > > >   ..,
> > > >   - faststream without backchannel
> > > >   - aptX
> > > >   - aptX-HD
> > > >   - aptX-LL without backannel
> > > >   - LDAC
> > > >   ...
> > > > * a2dp_sink
> > > >   - same as in a2dp_source
> > > > * a2dp_source_with_backchannel
> > > >   - faststream with backchannel
> > > >   - aptX-LL with backchannel
> > > > * a2dp_sink_with_backchannel
> > > >   - same as in a2dp_source_with_backchannel
> > > > 
> > > > So sub-profile would be one specific configuration of codec. In HSP/HFP
> > > > there would be two sub-profiles for every codec (one for software
> > > > encoding, one for hardware encoding). In A2DP it may be various. E.g.
> > > > for "SBC profiles" there would be one sub-profile for every SBC
> > > > configuration. For faststream codec there would be two, one with
> > > > backchannel, one without. Default sub-codec selection would be up to
> > > > the pulseaudio module. And applications (pactl / pavucontrol / ...)
> > > > would be able to change PA sub-profile in similar way how they can do it
> > > > for PA profiles.
> > > > 
> > > > Of course some sub-profiles does not have to be supported (e.g. mSBC in
> > > > hardware encoding) and in these cases pulseaudio would not announce
> > > > these unsupported sub-profiles.
> > > > 
> > > > What do you think about this my proposal?
> > > > 
> > > > Tanu, is this acceptable for you?
> > > > 
> > > > Basically it is needed only to add APIs for applications to:
> > > > * get current sub-profile of card
> > > > * get list of all sub-profiles for specific profile
> > > > * change sub-profile of card
> > > > 
> > > > I'm not saying how would API look like. I'm open for implementation
> > > > details... I can imagine that we can extend PA protocol or use messaging
> > > > API for it or anything else...
> > > > 
> > > > 
> > > > (*) - currently Linux kernel blocks this usage of mSBC HW encoding and
> > > > force userspace to do whole encoding at application level (in
> > > > pulseaudio).
> > > 
> > > Thanks for the proposal, I read it now and all the different use cases
> > > seem to fit quite neatly to the sub-profile model, so it's a good
> > > starting point at least.
> > > 
> > > I'm trying to think this from the user's point of view. The proposed UI
> > > seems to be that first the sub-profile is chosen (activated) and then
> > > the codec parameters can be chosen.
> > 
> > On Sunday 12 January 2020 09:23:52 Tanu Kaskinen wrote:
> > > I meant "first the profile is chosen", not sub-profile.
> > 
> > Yes, first user chose profile and then sub-profile (= codec parameters).
> > 
> > > I think it would be better to have
> > > all codec options visible regardless of the currently active profile.
> > > Some profiles aren't meant to be manually chosen (a2dp_source,
> > > headset_audio_gateway)
> > 
> > Bluetooth may support both a2dp_sink and a2dp_source profiles. And it
> > may support different set of A2DP codecs on these profiles. (E.g. it can
> > encode LDAC, but cannot decode LDAC).
> > 
> > So we cannot make visible all options as some of them would not make
> > sense.
> > 
> > We can e.g. put all codecs into "sub-profile" select box, but codecs not
> > "compatible" with currently selected profile would be gray-out and user
> > would not be able to choose it. But he would see that PA support this
> > option in different configuration. Or add some other GUI where user
> > choose codec (=sub-profile) and then another select-box would find all
> > compatible profiles for this codec, so he could choose what he want.
> > 
> > But this is just designing UI for selecting sub-profile. It does not
> > involve any changes on pulseaudio server.
> > 
> > > and it's probably good to avoid unnecessary
> > > manual changes between a2dp_sink and headset_head_unit too, because
> > > that may confuse or make things more complicated for the automatic
> > > profile switching logic.
> > 
> > We cannot avoid it. Choosing codec is part of connection establishment.
> > So if user is not satisfied with currently used codec he should be able
> > to change it.
> > 
> > > Setting the codec preferences and choosing the
> > > active profile would be good to keep as separate actions.
> > > 
> > > Are there differences in use cases for HFP and bidirectional A2DP? To
> > 
> > I guess you are asking for "bidirectional A2DP codec with activated
> > backchannel". As I saw, aptX-LL is often used without backchannel, so is
> > only one-directional.
> > 
> > And what are differences? HFP is synchronous with a low delay and
> > supports additional features, e.g. phone control, battery power status,
> > button press, vendor extensions, etc...
> > 
> > Bidirectional A2DP codec FastStream has better quality then HFP CVSD.
> > Some devices does not have to support HFP mSBC.
> > 
> > So user had to choose: Does it need better quality? Then choose A2DP.
> > Does it need to control phone calls? Then choose HFP with lower audio
> > quality.
> > 
> > I'm looking at it: You need to choose profile (HFP or A2DP) based on
> > what features you need. And then choose audio quality.
> > 
> > All bluetooth audio devices describes which bluetooth profiles (HFP,
> > A2DP, ...) they support and also what functionality they in their
> > profile they support. So user needs to know what is he using to not be
> > surprised that something does not work (just because PA decided to use
> > other bluetooth profile).
> > 
> > > me it seems that the use cases are the same, and the user should have
> > > the HFP and bidirectional A2DP codecs in one list (not necessarily in
> > > this order):
> > > 
> > > * CVSD (hardware)
> > > * mSBC (software)
> > > * mSBC (hardware)
> > > * AuriStream
> > > * FastStream
> > > * aptX-LL
> > > 
> > > So I'm not sure the new bidirectional A2DP profiles are needed, but
> > > maybe having them makes the implementation easier. I'd be ok with
> > > adding the new profiles, but making the primary codec configuration API
> > > separate and automatically switching the profile when the preferred
> > > bidirectional codec is changed.
> > > 
> > > I noticed you listed "aptX-LL without backchannel" as a sub-profile.
> > > Isn't that the same as plain aptX?
> > 
> > No. Those are two different things. aptX-LL should have lower latency.
> > Encoded stream is same, but packed differently at lower bluetooth
> > layers (different buffer sizes and maybe on receiver side also some
> > optimizations).
> > 
> > > Would you be willing to document the codec negotiation process in the
> > > wiki?
> > 
> > What exactly?
> 
> The things that you mentioned below:
>  * Either side can establish the connection, so it's not obvious what
> things can trigger a connection. For example, what might cause a
> headset to initiate the connection?
>  * BlueZ may decide the codec instead of PulseAudio. What logic does it
> use?
>  * There are many places with their own defaults. What are these places
> and their codec selection logic?
> 
> Maybe these things doesn't affect the UI as I first thought, but it
> would be great to have a reference anyway, so that the system can be
> understood when debugging codec issues.

Ok. So after implementation of multicodec support would be in
pulseaudio I can document how it works and how it can be configured.

> > > I can give you a wiki account. I believe you have already
> > > explained this before in emails, but I've forgotten the details. To me
> > > it would make sense if only the playback side (or the audio gateway for
> > > bidirectional modes) could decide the codec, but I recall the reality
> > > is more complicated, and there were multiple places where the chosen
> > > codec may be remembered.
> > 
> > Side which establishing connection choose codec. It does not have to be
> > playback side. And connection establishment is done by calling DBus
> > bluez function, independently of pulseaudio (in most cases desktop
> > bluetooth GUI application do that). So we cannot avoid it.
> > 
> > But there are many places which has own defaults. So establishing
> > connection would work without specifying codec and some remembered /
> > default value would be used.
> > 
> > > The negotiation details affect the constraints
> > > for UI design.
> > 
> > We can do this: When user choose profile, then PA issue command to
> > activate profile with some default codec. User would see in GUI what was
> > chosen and could change if is not happy with it. If he does not care
> > about it he would use default / remembered option and so does not have
> > to choose anything.
> 
> Here's my current overall thinking:
> 
> Having the six profiles as you suggested is good.
> 
> If I understood correctly, you're proposing to make the sub-profiles a
> generic concept rather than a bluetooth specific API. I'm a bit
> hesitant to add an abstraction that doesn't in reality abstract
> anything, because bluetooth is the only user for now and other users
> haven't been suggested. On the other hand, I don't foresee concrete
> problems with this either, so I guess it doesn't really matter.

Ok. If it simplify things, we can decide that subprofile implementation
would be in bluetooth pa module and therefore bluetooth specific.

Now the remaining question is: Which API or solution would be used to
implement subprofiles (codec selection) by GUI/CLI tools? Pulseaudio
client <--> server protocol would be needed to extended to support this
kind of subprofile commands. Or messaging API (still not merged) could
be used for this.

> In my previous mail I suggested keeping the codec preference
> configuration independent from the active profile, but this can be
> implemented later if this is something people want.

-- 
Pali Rohár
pali.rohar at gmail.com