[pulseaudio-discuss] About extra A2DP codecs support in bluetooth module

[Arun Raghavan]

On Thu, 20 Jul 2017, at 06:12 AM, Qu Wenruo wrote:
> 
> 
> On 2017年07月20日 04:13, Luiz Augusto von Dentz wrote:
> > Hi All,
> > 
> > On Sat, Jul 15, 2017 at 9:08 AM, Arun Raghavan <arun at arunraghavan.net> wrote:
> >>
> >>
> >> On Sat, 15 Jul 2017, at 11:37 AM, Arun Raghavan wrote:
> >>>
> >>>
> >>> On Fri, 14 Jul 2017, at 06:50 AM, Qu Wenruo wrote:
> >>>>
> >>>>
> >>>> On 2017年07月12日 17:57, Arun Raghavan wrote:
> >>>>> On Fri, 7 Jul 2017, at 08:35 PM, Tanu Kaskinen wrote:
> >>>>>> On Fri, 2017-07-07 at 20:33 +0800, Qu Wenruo wrote:
> >>>>>>> After a quick glance into the code (without much knowledge about
> >>>>>>> pulseaudio), I found that pulseaudio is just using sbc library to do the
> >>>>>>> encode.
> >>>>>>>
> >>>>>>>    From a2dp_process_render():
> >>>>>>> ---
> >>>>>>>        while (PA_LIKELY(to_encode > 0 && to_write > 0)) {
> >>>>>>>            ssize_t written;
> >>>>>>>            ssize_t encoded;
> >>>>>>>
> >>>>>>>            encoded = sbc_encode(&sbc_info->sbc,
> >>>>>>>                                 p, to_encode,
> >>>>>>>                                 d, to_write,
> >>>>>>>                                 &written);
> >>>>>>> ---
> >>>>>>>
> >>>>>>> So there is really nothing blocking us to implement other codec.
> >>>>>>> For AAC codec, just (well, without tons of preparation and setup) call
> >>>>>>> faacEncEncode() will be the core part.
> >>>>>>> Copyright sh*t will only restrict the related library, not the PA module.
> >>>>>>> (So if we could create a aptX codec library, then it will be possible to
> >>>>>>> support)
> >>>>>>>
> >>>>>>> While the really hard part would be the preparation part, including
> >>>>>>> creating a structure for faac encoder to contain a faacEncHandle and
> >>>>>>> other needed info from sample rate to profile, just like sbc_info_t.
> >>>>>>>
> >>>>>>> Although I have a basic idea of what to do, I'm still figuring out how
> >>>>>>> to handle all the details.
> >>>>>>> Like how to create an endpoint for AAC codec (codec 0 is registered at
> >>>>>>> register_endpoint, but shouldn't it be A2DP_CODEC_SBC instead of
> >>>>>>> intermediate number 0?)
> >>>>>>
> >>>>>> I don't know bluetooth details enough to answer. I'll add Luiz to Cc in
> >>>>>> case he knows. You could try asking on the bluez mailing list too.
> >>>>>
> >>>>> I would suggest just hiding away the entire RTP payloading and encoding
> >>>>> using GStreamer here. That neatly sidesteps the issue of
> >>>>> hardware/software codecs, IP-sensitive codec libraries, and so forth. We
> >>>>> probably want to keep the SBC path available the way it is right now to
> >>>>> avoid GStreamer as a hard-dep, but otherwise, I think that's the more
> >>>>> sensible approach to this.
> >>>>
> >>>> I'm still fighting against dbus and bluez5 things, so GStreamer is not
> >>>> my primary goal.
> >>>> But the idea to let a framework to handle everything indeed looks neat
> >>>> and clean.
> >>>>
> >>>> However I'm more concerned about how the final A2DP profile is
> >>>> determined.
> >>>>
> >>>>   From what I can see, pulseaudio bluetooth modules just register
> >>>> endpoint for bluez, with Codec, UUID and codec specified capabilities.
> >>>> However I didn't see how codec selection is done.
> >>>>
> >>>> Is it done by bluez5 or pulseaudio?
> >>>>
> >>>> My currect understanding to implement a new codec will need:
> >>>> 1) Register a new codec in register_endpoint() of bluez5-util.c of PA.
> >>>>      Instead of codec 0 (shouldn't it be A2DP_CODEC_SBC?), we must also
> >>>> register codec
> >>>>      A2DP_CODEC_MPEG24 with a2dp_aac_t as capability.
> >>>>
> >>>>     Well, updated a2dp-codec.h header will be needed, as there is no
> >>>> a2dp_aac_t in PA.
> >>>
> >>> Correct.
> >>>
> >>>> 2) Handle codec capabilities negotiation
> >>>>      endpoint_set_configuration() seems to be responsible to send out the
> >>>> final
> >>>>      SetConfiguration AVDTP packet.
> >>>>      But which codec will bluez5 choose? Just highest codec number of
> >>>> both SINK and SOURCE
> >>>>      device?
> >>>>
> >>>>      endpoint_select_configuration() seems to be related to handle the
> >>>> response from the SINK
> >>>>      device.
> >>>
> >>> Right again.
> >>>
> >>> Maybe get this working, and then we can worry about how to select
> >>> codecs. It'll probably have to be based on some module configuration
> >>> (which in turn will be guided by the h/w that this is running on).
> >>> Default could be something we select based on order of quality.
> >>>
> >>>>      But for multi-codec support, how do we distinguish one codec
> >>>> capability from another?
> >>>>
> >>>> 3) Record final codec profile into some structure of userdata in
> >>>> module-bluez5-devices.c
> >>>>
> >>>> 4) Call codec encode function in thread_func()
> >>>>
> >>>> Any comment is welcomed.
> >>>
> >>> With GStreamer, either you'd probably do something like appsrc !
> >>> <encoder> ! <rtp payloader> ! appsink to run the encode, in
> >>> a2dp_process_render().
> >>
> >> The "either" was for a second option involving directly writing to an
> >> fdsink, but I think we should keep that in PA so coexisting with the
> >> current SBC code is easier.
> > 
> > While this is all possible since BlueZ, nor most of the stacks,
> > supports the so called multiplex mode using computing intensive codecs
> > is not desirable, in fact the only reason to support AAC and MP3
> > probably is in a passthrough mode where the source just send directly
> > the data without any transcoding, otherwise we would need to encode
> > all audio using those codecs, including system alerts, etc, which may
> > not be very nice for battery powered devices.
> 
> Yes, extra encoding will reduce battery time of course.
> However IIRC iOS is using AAC codec if sink device support it, I think 
> it's not without any reason.

This would be a (fairly complex) next step, that imo we need in PA
anyway -- support for compressed formats that are not
IEC61937-payloaded.

-- Arun