[pulseaudio-discuss] GSoC: Call for project ideas

[Toby St Clere Smithe]

Hi Arun,

Thanks for your message. I'm going to reply to you first, and then reply
to David.

Arun Raghavan <arun.raghavan at collabora.co.uk> writes:
> Having means of doing non-PCM streaming would definitely be desirable.
> That said, though, I'm a bit wary of introducing codec and RTP
> dependencies in  PulseAudio (currently we have this at only one point,
> which is the Bluetooth modules - something I don't see a way around).
>
> Now there are two main concerns:
>
> 1. Codecs: choosing a codec is not simple. There are always potential
> reasons to look at one vs. the other - CPU utilisation, bandwidth, codec
> latency, quality, specific implementation (libav vs. reference
> implementation vs. hardware acceleration) and so on.

Indeed. I think a codec-agnostic implementation is important, as I
mentioned in an earlier message. I chose Opus for the same reasons as it
was designed for, and I appreciate the desire for some lossless codec
too. But I hadn't spent much time thinking about implementation
differences, mainly because at the time I did my research, there was
only one implementation!

Your ideas about using GStreamer are interesting, but I don't know much
about how a GStreamer pipeline would fit into PulseAudio's pipeline, and
how this would affect things like latency. Nonetheless, you're certainly
right about the maintenance burden, and I am a fan of GStreamer in any
case. To have its power in PulseAudio would be very interesting, and
certainly worth the research I would need to do to understand (say) the
effect on latency.

> 2. RTP: as our usage gets more complicated, we're going to end up
> implementing and maintaining a non-trivial RTP stack, which is actually
> quite hard.
>
> Deciding where to draw the line with regards to what does and does not
> belong in PulseAudio is a bit tricky, but in my mind, encoding/decoding
> should very much not be in PulseAudio because that beast inevitably gets
> more complicated as you try to do more, and there are others solving the
> problem system-wide.
>
> RTP, I can see a case for it being in PulseAudio, but it is also
> complicated, and as with codecs, there are other places in the system
> where it gets more attention and maintenance.

David asked in his other messages about whether my plans were for RTP or
for the native protocol. I'll describe my thinking here, too.

Originally, I had looked at writing code for implementing these codecs
in the native protocol, because I felt that I wanted to build the
support as closely to the core of PulseAudio as possible; doing so
seemed to be the easiest way to keep on top of the logic for adjusting
bitrate etc according to latency and frame drops, and I also felt that
doing so in the proprietary protocol would give us more coding liberty.

Now, however, I am largely in agreement that it is probably foolish to
maintain this whole stack in PulseAudio, and furthermore, it's probably
good to use the streaming protocols that are widely known and
understood, rather than duplicating the effort in a way that is likely
to be suboptimal.

Do you agree that using RTP rather than building the GStreamer pipeline
closer to the core of PulseAudio is probably the best plan?

> The simplest idea I can think of to deal with this meaningfully is to
> wrap a sink/source around a GStreamer pipeline to offload all that work
> that we don't want to duplicate in PulseAudio .

Quite.

> On the sink side, we'd hook up to an appsrc to feed PCM data to a
> pipeline. The pipeline would take care of encoding, RTP packetisation
> and possibly a corresponding RTCP stream. This would allow codec
> selection to be flexible, and in the distant future, could even support
> taking encoded data directly.
>
> On the source side, we'd hook up to an appsink, receiving PCM data from
> the pipeline. The pipeline would take care decoding whatever the format
> is, take care of RTCP and maybe more advanced features such as a jitter
> buffer and packet-loss concealment (all of this can be plugged in or
> not, depending on configuration).

This does sound like a good plan to solve the given problem, save for my
concerns about monitoring and integration above. My other thought, which
would not work in the context of using an appsrc/appsink, is that having
GStreamer built more closely into PulseAudio would also mean we could
offload things like resampling (which I noticed was discussed in another
GSoC-related thread) to the GStreamer pipeline, rather than have to
maintain those code-paths, too.

> Doing it this way means you're using a better RTP stack that gets
> attention from a number of other use cases (plus assorted related
> goodies) and support for multiple codecs.

Right, indeed. But I'm still not entirely sure about RTP vs native, and
where the GStreamer code should go. My temptation is to couple
PulseAudio as closely as possible to GStreamer. But I thought about this
a little when doing my initial research, and came to the conclusion that
if this was a good idea, it would have been done already. What am I
missing?

Indeed, if we build GStreamer into pulsecore,  then we could use RTP or
the native protocol, as we saw fit. If we went for the appsrc/sink
solution, then we'd be less flexible.

I haven't read the code in depth in a while, and I have exams soon. I
cannot start my research properly until they are out of the way in a
month or so, so I don't know how plausible this is. But it sounds quite
fun.


Cheers,

Toby