[pulseaudio-discuss] a question about audio synchronization between local sink and tunnel sink

[Tanu Kaskinen]

On Thu, 2012-10-25 at 23:50 +0000, Sun, Xiaodong wrote:
> Tanu,
> 
> I have modified code like this in module-combine-sink.c after
> exchanging idea with you.
> 
> In function sink_input_pop_cb():  add these lines before calling
> request_memblock()
>     ...
> 
>     if (o->just_attached) {
>         pa_usec_t sink_input_latency, sink_latency, total_latency1, total_latency2;
>         struct output *j;
> 
>         sink_input_latency = pa_sink_input_get_latency_within_thread(i, &sink_latency);
>         sink_input_latency += sink_latency;
>         total_latency1 = sink_input_latency + pa_bytes_to_usec(pa_memblockq_get_length(o->memblockq), &o->sink_input->sample_spec);
>         pa_log_warn("[%s] latency %0.2fms, output buffer latency %0.2fms.", o->sink_input->sink->name, (double)total_latency1/PA_USEC_PER_MSEC, (double)(total_latency1-sink_input_latency)/PA_USEC_PER_MSEC);
> 
>         /* OK, let's send this data to the other threads */
>         PA_LLIST_FOREACH(j, o->userdata->thread_info.active_outputs) {
>             if (j != o) {
>                 sink_input_latency = pa_sink_input_get_latency_within_thread(j->sink_input, &sink_latency);
>                 sink_input_latency += sink_latency;
>                 total_latency2 = sink_input_latency + pa_bytes_to_usec(pa_memblockq_get_length(j->memblockq), &j->sink_input->sample_spec);
>                 pa_log_warn("Peer: [%s] latency %0.2fms, output buffer latency %0.2fms.", j->sink_input->sink->name, (double)total_latency2/PA_USEC_PER_MSEC, (double)(total_latency2-sink_input_latency)/PA_USEC_PER_MSEC);
>                 if (total_latency1 > total_latency2)
>                     pa_memblockq_drop(o->memblockq, pa_usec_to_bytes(total_latency1-total_latency2, &o->sink_input->sample_spec));
>                 else if (total_latency1 < total_latency2)
>                     pa_memblockq_rewind(o->memblockq, pa_usec_to_bytes(total_latency2-total_latency1, &o->sink_input->sample_spec));
>                 break;
>             }
>         }

This code is not thread-safe, but maybe you already knew that - I guess
the code is ok for initial experiments. It certainly revealed nicely the
problem with the initial latency of the tunnel sink.

Also, the algorithm doesn't seem very reliable: it assumes that the last
written chunk in both memblockqs is the same, and that both outputs have
received all chunks since the beginning (not true if one output is
created later the other, but I guess in your case they are created at
the same time so that the first chunk always is sent to both outputs).

> 
>         o->just_attached = 0;
>     }
> 
> In function sink_input_attach_cb(): add  o->just_attached = 1;
> 
> In function sink_input_detach_cb(): add  o->just_attached = 0;
> 
> 
> Then I got following logs when playing audio. Of course
> un-synchronization issue is still there.
> 
> W: [module-tunnel] module-combine-sink.c: [tunnel_sink] attached
> W: [module-tunnel] module-combine-sink.c: [tunnel_sink] latency 0.00ms, output buffer latency 0.00ms.
> W: [alsa-sink] module-combine-sink.c: [alsa_output] attached
> W: [alsa-sink] module-combine-sink.c: [alsa_output] latency 0.88ms, output buffer latency 0.00ms.
> W: [alsa-sink] module-combine-sink.c: Peer: [tunnel_sink] latency 0.00ms, output buffer latency 0.00ms.
> W: [pulseaudio] module-combine-sink.c: [alsa_output] sample rates too different, not adjusting (44100 vs. 33380).
> W: [pulseaudio] module-combine-sink.c: [alsa_output] sample rates too different, not adjusting (44100 vs. 28907).
> W: [pulseaudio] module-combine-sink.c: [alsa_output] sample rates too different, not adjusting (44100 vs. 31788).
> 
> From these logs, we can tell tunnel sink's sink-input is attached
> first and its pop_cb function is called before alsa sink's sink-input
> is attached. When alsa sink's sink-input is attached and its pop_cb
> function is called, the tunnel sink's latency is still 0. It could be
> because the gap between alsa sink attachment and tunnel sink attached
> is too small and tunnel sink's buffer is still empty. The latency at
> this time doesn't show the real latency that sink has. So this method
> doesn't work. Actually this method only works when the other attached
> sinks are already in stable playback status, meaning its latency is
> constantly stable, before new sink is attached.

Even if the tunnel sink's buffer is "empty", its true latency isn't
zero. So, pa_sink_input_get_latency_within_thread() returns incorrect
values. I guess the latency information just isn't there immediately
after the tunnel sink has become active (either created or unsuspended).
I'm not sure what would be the best way to address this - maybe
send_data() in module-tunnel.c should be modified to generate silence
instead of calling pa_sink_render() until the latency information
becomes available. The SINK_MESSAGE_UPDATE_LATENCY handler in
module-tunnel.c seems like the right place to flag that now the latency
information is available.

> For my previous questions about sink and sink-input. My intention is
> to ask what is the difference between sink and sink-input. I saw both
> of them are actually object instances which can register and accept
> messages. After spending more time on reading the code, I seems to
> know the difference now. Sink is actually a real/virtual sink module
> instance which normally has a separate thread attached and can poll
> and process async messages to it. While sink-input is corresponding
> sink's representative at the other end of buffer. Is my understanding
> correct?

The description for "sink" sounds quite right, but the "sink-input"
description isn't so accurate (or I just don't get what you mean).

I'll try an explanation: sink is an "output device". Sink input is a
"playback stream" (so it's not a "representative" of the sink, it's a
representative of whatever wants to send audio to the sink, in your case
the sink inputs represent outputs of the combine sink). Any number of
sink inputs can be connected to a sink. The sink mixes the inputs and
sends the mixed audio to some backend. An alsa sink sends the audio to
alsa, a tunnel sink sends the audio to another pulseaudio instance over
the network and a combine sink sends the audio to multiple other sinks.

The clock in the system is provided by the sink: sink inputs can't just
push data to a sink, it's always so that a sink pulls audio from the
sink inputs when the backend needs more data.

The audio processing happens in a separate thread ("IO thread"), which
is created by the sink. So, the audio processing code of sink inputs
runs in the IO thread of the sink that the input is connected to. Each
sink has has its own IO thread (with the exception of filter sinks, but
that's not relevant here). In addition to the IO threads, there's the
"main thread", i.e. the thread where pulseaudio's main() function runs,
and where the system control is done.

The main thread communicates with the IO threads with messages, sent
with pa_asyncmsgq_send() (synchronous) and pa_asyncmsgq_post()
(asynchronous). In case of module-combine-sink, message passing is used
also to communicate between the combine sink's IO thread and the IO
threads of the sinks that the combine sink is connected to.

-- 
Tanu