[pulseaudio-discuss] Working on a new network transport for PulseAudio

[Tanu Kaskinen]

Sorry for the slow reply!

I commented on a couple of points below. I didn't really have any
comments for the rest. I mainly just wanted you to be aware of all the
complexities that you may want to deal with in the future, and it seems
that you're willing to deal with pretty much all of it, which is nice.


On Wed, 2019-05-15 at 09:31 +0200, Victor Gaydov wrote:
> On Fri, 10 May 2019 21:15:48 +0300
> Tanu Kaskinen <tanuk at iki.fi> wrote:
> 
> > On Mon, 2019-05-06 at 21:35 +0400, Victor Gaydov wrote:
> > > Hi,
> > > 
> > > I'm working on Roc, a new project implementing real-time streaming
> > > over network[1].
> > > 
> > > Currently it can stream PCM using RTP + FECFRAME (Reed-Solomon and
> > > LDPC-Staircase are supported, using OpenFEC library). There are
> > > plans to add support for more encodings (including Opus) and
> > > protocols (including SAP/SDP).
> > > 
> > > Among other things, we've implemented a couple of PulseAudio modules
> > > that use Roc as a network transport. I've written an overview post
> > > about the project and these modules[2].
> > > 
> > > Both Roc and PA modules are in early stage of development. PA
> > > modules still miss some essential features, e.g. there is no
> > > latency reporting and service discovery.
> > > 
> > > I'd like to discuss the opportunity of submitting those modules to
> > > the PA upstream in future. Some thoughts?
> > > 
> > > [1] https://github.com/roc-project/roc
> > > [2] https://gavv.github.io/articles/new-network-transport/
> > > 
> > > -- Victor  
> > 
> > Miscellaneous thoughts in random order:
> > 
> > At least I'm in favor of accepting the new modules in PulseAudio, if
> > they provide less glitches than the tunnel modules or the existing RTP
> > modules. And I'd love to have the adaptive resampling logic maintained
> > in an external library by someone who knows how it works (I don't
> > understand and I don't look forward to trying to learn how the
> > resampling code in the existing RTP modules works).
> 
> Great!
> 
> > I should probably mention that if you submit patches, I can't promise
> > a timely review - there's too much work that I'm already committed to.
> 
> OK.
> 
> > Arun has in the past talked about some plans for reworking the RTP
> > stuff to use GStreamer - I hope that won't be an impediment for
> > accepting these modules.
> 
> Interesting. This should be discussed, I think, because Roc
> features overlap with the network part of GStreamer.
> 
> > Are there any plans to support multicast? I don't know how important
> > that is to people, it's just that I think the existing RTP modules use
> > multicast by default (which tends to break wireless networks...)
> 
> Multicast is in our roadmap, but I'd like to work on it a bit later,
> after implementing some more basic stuff like receiver feedback,
> session negotiation, and compression.
> 
> > Can one receiver connect to multiple senders? If there are two client
> > machines that want to use speakers connected to a single receiver
> > machine, does module-roc-sink-input have to be loaded twice (and with
> > different ports)?
> 
> Single receiver handles multiple senders connected to it (and mixes
> their streams).

Is there only one sink input object even if there are multiple senders?
So the senders can't be routed independently to different speakers for
example? I wonder how module-rtp-recv deals with multiple senders...
Hmm, there seems to be one sink input per session struct, and I guess
there's one session per sender?

> > Any plans for dynamic latency based on stream requests? If the goal is
> > to support low latency, it's probably desirable to have a bit larger
> > latency for music playback, and reduce the latency on the fly when a
> > low-latency application starts.
> 
> There are plans to implement automatic latency adjustment (depending on
> the network conditions) on the receiver side.
> 
> Allowing the user to adjust the latency manually on the sender side
> sounds reasonable but we need to think what protocols are suited for
> that. Probably RTSP, which is in our roadmap.
> 
> > How much of the parameters could theoretically be changed on the fly
> > (I say theoretically, because you probably haven't implemented any
> > such controls yet)? Imagine a GUI for configuring this system - trying
> > different resampler profiles or latency settings on the fly could be
> > useful. Changing the IP address or port probably doesn't need to be
> > doable mid-stream.
> 
> Currently it's not possible to change anything on fly, except binding
> new ports.
> 
> There are plans to implement automatic adjustment of latency, FEC block
> size, packet size, and bitrate. This means that in future all these
> parameters will be configurable on fly at least internally. And so it
> would be possible to make them configurable via API too.
> 
> > People have requested in the past that the tunnel sink would change
> > the stream sample rate and format on the fly based on the
> > application's stream, similar to how the hardware sink supports
> > automatic sample rate switching. Would this be doable? It doesn't
> > need to (and indeed shouldn't) be done mid-stream, but if nothing is
> > currently using the Roc sink and a new stream appears, it would be
> > good to configure the network stream parameters based on the
> > application stream.
> 
> I think it would be possible after we'll add some sort of session
> negotiation (RTSP or SIP). I'm planning to start working on it soon.
> 
> > The blog post has this sentence: "Roc performs the clock rate
> > adjustment on the receiver, while PulseAudio does it on the sender" -
> > that's not true. Doing the adjustment on the sender would require some
> > kind of feedback system from the receiver to the sender, and that
> > doesn't exist. PulseAudio does the adjustment on the receiver.
> 
> My bad. Fixed.
> 
> > What happens if the connection dies temporarily (receiver reboots or
> > crashes, or there's a network outage)? Can the system automatically
> > recover when the receiver comes back online?
> 
> Currently things work very simple. Sender just sends stream to the
> given address and has no feedback from the receiver. Receiver
> automatically creates a session when it gets a packet from a new sender
> and removes the session when there are no packets during a timeout. So
> yes, the system will automatically recover.
> 
> Things will become a bit more complicated when we'll implement session
> negotiation. We'll have to re-negotiate the session after an outage.
> 
> > Can the system deal with the hardware sink changing on the fly? There
> > will be a discontinuity in the reported latency. Similarly, can the
> > system deal with the user changing the latency offset of a port? That
> > too will cause a strange jump in latency reports.
> 
> Currently no (I didn't test it yet).

Supporting at least the sink change case seems pretty important to me.

Another situation where there can be latency discontinuities is when
there's an underrun in the hardware sink.

> > Can the system deal with the hardware sink suspending? The Roc sink
> > input will be connected to the hardware sink, but the sink won't be
> > asking for more data until the sink resumes.
> 
> Currently no (I have tested it; sink suspending breaks the receiver).
> 
> > Based on the great documentation, Roc looks like a quality product!
> 
> Thanks :)

-- 
Tanu

https://www.patreon.com/tanuk
https://liberapay.com/tanuk