[pulseaudio-discuss] About extra A2DP codecs support in bluetooth module

[Luiz Augusto von Dentz]

Hi All,

On Sat, Jul 15, 2017 at 9:08 AM, Arun Raghavan <arun at arunraghavan.net> wrote:
>
>
> On Sat, 15 Jul 2017, at 11:37 AM, Arun Raghavan wrote:
>>
>>
>> On Fri, 14 Jul 2017, at 06:50 AM, Qu Wenruo wrote:
>> >
>> >
>> > On 2017年07月12日 17:57, Arun Raghavan wrote:
>> > > On Fri, 7 Jul 2017, at 08:35 PM, Tanu Kaskinen wrote:
>> > >> On Fri, 2017-07-07 at 20:33 +0800, Qu Wenruo wrote:
>> > >>> After a quick glance into the code (without much knowledge about
>> > >>> pulseaudio), I found that pulseaudio is just using sbc library to do the
>> > >>> encode.
>> > >>>
>> > >>>   From a2dp_process_render():
>> > >>> ---
>> > >>>       while (PA_LIKELY(to_encode > 0 && to_write > 0)) {
>> > >>>           ssize_t written;
>> > >>>           ssize_t encoded;
>> > >>>
>> > >>>           encoded = sbc_encode(&sbc_info->sbc,
>> > >>>                                p, to_encode,
>> > >>>                                d, to_write,
>> > >>>                                &written);
>> > >>> ---
>> > >>>
>> > >>> So there is really nothing blocking us to implement other codec.
>> > >>> For AAC codec, just (well, without tons of preparation and setup) call
>> > >>> faacEncEncode() will be the core part.
>> > >>> Copyright sh*t will only restrict the related library, not the PA module.
>> > >>> (So if we could create a aptX codec library, then it will be possible to
>> > >>> support)
>> > >>>
>> > >>> While the really hard part would be the preparation part, including
>> > >>> creating a structure for faac encoder to contain a faacEncHandle and
>> > >>> other needed info from sample rate to profile, just like sbc_info_t.
>> > >>>
>> > >>> Although I have a basic idea of what to do, I'm still figuring out how
>> > >>> to handle all the details.
>> > >>> Like how to create an endpoint for AAC codec (codec 0 is registered at
>> > >>> register_endpoint, but shouldn't it be A2DP_CODEC_SBC instead of
>> > >>> intermediate number 0?)
>> > >>
>> > >> I don't know bluetooth details enough to answer. I'll add Luiz to Cc in
>> > >> case he knows. You could try asking on the bluez mailing list too.
>> > >
>> > > I would suggest just hiding away the entire RTP payloading and encoding
>> > > using GStreamer here. That neatly sidesteps the issue of
>> > > hardware/software codecs, IP-sensitive codec libraries, and so forth. We
>> > > probably want to keep the SBC path available the way it is right now to
>> > > avoid GStreamer as a hard-dep, but otherwise, I think that's the more
>> > > sensible approach to this.
>> >
>> > I'm still fighting against dbus and bluez5 things, so GStreamer is not
>> > my primary goal.
>> > But the idea to let a framework to handle everything indeed looks neat
>> > and clean.
>> >
>> > However I'm more concerned about how the final A2DP profile is
>> > determined.
>> >
>> >  From what I can see, pulseaudio bluetooth modules just register
>> > endpoint for bluez, with Codec, UUID and codec specified capabilities.
>> > However I didn't see how codec selection is done.
>> >
>> > Is it done by bluez5 or pulseaudio?
>> >
>> > My currect understanding to implement a new codec will need:
>> > 1) Register a new codec in register_endpoint() of bluez5-util.c of PA.
>> >     Instead of codec 0 (shouldn't it be A2DP_CODEC_SBC?), we must also
>> > register codec
>> >     A2DP_CODEC_MPEG24 with a2dp_aac_t as capability.
>> >
>> >    Well, updated a2dp-codec.h header will be needed, as there is no
>> > a2dp_aac_t in PA.
>>
>> Correct.
>>
>> > 2) Handle codec capabilities negotiation
>> >     endpoint_set_configuration() seems to be responsible to send out the
>> > final
>> >     SetConfiguration AVDTP packet.
>> >     But which codec will bluez5 choose? Just highest codec number of
>> > both SINK and SOURCE
>> >     device?
>> >
>> >     endpoint_select_configuration() seems to be related to handle the
>> > response from the SINK
>> >     device.
>>
>> Right again.
>>
>> Maybe get this working, and then we can worry about how to select
>> codecs. It'll probably have to be based on some module configuration
>> (which in turn will be guided by the h/w that this is running on).
>> Default could be something we select based on order of quality.
>>
>> >     But for multi-codec support, how do we distinguish one codec
>> > capability from another?
>> >
>> > 3) Record final codec profile into some structure of userdata in
>> > module-bluez5-devices.c
>> >
>> > 4) Call codec encode function in thread_func()
>> >
>> > Any comment is welcomed.
>>
>> With GStreamer, either you'd probably do something like appsrc !
>> <encoder> ! <rtp payloader> ! appsink to run the encode, in
>> a2dp_process_render().
>
> The "either" was for a second option involving directly writing to an
> fdsink, but I think we should keep that in PA so coexisting with the
> current SBC code is easier.

While this is all possible since BlueZ, nor most of the stacks,
supports the so called multiplex mode using computing intensive codecs
is not desirable, in fact the only reason to support AAC and MP3
probably is in a passthrough mode where the source just send directly
the data without any transcoding, otherwise we would need to encode
all audio using those codecs, including system alerts, etc, which may
not be very nice for battery powered devices.

If someone would like a good alternative to SBC I would suggest using
OPUS, since that is used for VoIP and webrtc the codec should be
probably less computing intensive and it most likely more suitable for
live streams. gstreamer supports OPUS so it would probably make sense
to use it. Obviously, this would only work if both ends support OPUS,
and the stream initiator actually selects it to be used over SBC.

There is one last problem, despite some Android phones claiming to
have codec preference in practice there is no way to influence the
codec selection of the remote end since both ends can start a new
stream and over A2DP one can just enumerate codec capabilities. In
BlueZ we just assume the order of endpoints registration shall be used
as priority/preference, though we do this per process/D-Bus
connection.


-- 
Luiz Augusto von Dentz