[pulseaudio-discuss] a question about audio synchronization between local sink and tunnel sink

[Sun, Xiaodong]

Tanu,

Thanks for your detailed explanation of sink-input. I am clear of it now.
In term of the solution, what else would you like to suggest me to try from system point of view? It seems only changing module-combine-sink module is not a feasible solution. 

- Xiaodong

-----Original Message-----
From: Tanu Kaskinen [mailto:tanuk at iki.fi] 
Sent: Friday, October 26, 2012 2:49 AM
To: Sun, Xiaodong
Cc: pulseaudio-discuss at lists.freedesktop.org
Subject: Re: [pulseaudio-discuss] a question about audio synchronization between local sink and tunnel sink

On Thu, 2012-10-25 at 23:50 +0000, Sun, Xiaodong wrote:
> Tanu,
> 
> I have modified code like this in module-combine-sink.c after 
> exchanging idea with you.
> 
> In function sink_input_pop_cb():  add these lines before calling
> request_memblock()
>     ...
> 
>     if (o->just_attached) {
>         pa_usec_t sink_input_latency, sink_latency, total_latency1, total_latency2;
>         struct output *j;
> 
>         sink_input_latency = pa_sink_input_get_latency_within_thread(i, &sink_latency);
>         sink_input_latency += sink_latency;
>         total_latency1 = sink_input_latency + pa_bytes_to_usec(pa_memblockq_get_length(o->memblockq), &o->sink_input->sample_spec);
>         pa_log_warn("[%s] latency %0.2fms, output buffer latency 
> %0.2fms.", o->sink_input->sink->name, 
> (double)total_latency1/PA_USEC_PER_MSEC, 
> (double)(total_latency1-sink_input_latency)/PA_USEC_PER_MSEC);
> 
>         /* OK, let's send this data to the other threads */
>         PA_LLIST_FOREACH(j, o->userdata->thread_info.active_outputs) {
>             if (j != o) {
>                 sink_input_latency = pa_sink_input_get_latency_within_thread(j->sink_input, &sink_latency);
>                 sink_input_latency += sink_latency;
>                 total_latency2 = sink_input_latency + pa_bytes_to_usec(pa_memblockq_get_length(j->memblockq), &j->sink_input->sample_spec);
>                 pa_log_warn("Peer: [%s] latency %0.2fms, output buffer latency %0.2fms.", j->sink_input->sink->name, (double)total_latency2/PA_USEC_PER_MSEC, (double)(total_latency2-sink_input_latency)/PA_USEC_PER_MSEC);
>                 if (total_latency1 > total_latency2)
>                     pa_memblockq_drop(o->memblockq, pa_usec_to_bytes(total_latency1-total_latency2, &o->sink_input->sample_spec));
>                 else if (total_latency1 < total_latency2)
>                     pa_memblockq_rewind(o->memblockq, pa_usec_to_bytes(total_latency2-total_latency1, &o->sink_input->sample_spec));
>                 break;
>             }
>         }

This code is not thread-safe, but maybe you already knew that - I guess the code is ok for initial experiments. It certainly revealed nicely the problem with the initial latency of the tunnel sink.

Also, the algorithm doesn't seem very reliable: it assumes that the last written chunk in both memblockqs is the same, and that both outputs have received all chunks since the beginning (not true if one output is created later the other, but I guess in your case they are created at the same time so that the first chunk always is sent to both outputs).

> 
>         o->just_attached = 0;
>     }
> 
> In function sink_input_attach_cb(): add  o->just_attached = 1;
> 
> In function sink_input_detach_cb(): add  o->just_attached = 0;
> 
> 
> Then I got following logs when playing audio. Of course 
> un-synchronization issue is still there.
> 
> W: [module-tunnel] module-combine-sink.c: [tunnel_sink] attached
> W: [module-tunnel] module-combine-sink.c: [tunnel_sink] latency 0.00ms, output buffer latency 0.00ms.
> W: [alsa-sink] module-combine-sink.c: [alsa_output] attached
> W: [alsa-sink] module-combine-sink.c: [alsa_output] latency 0.88ms, output buffer latency 0.00ms.
> W: [alsa-sink] module-combine-sink.c: Peer: [tunnel_sink] latency 0.00ms, output buffer latency 0.00ms.
> W: [pulseaudio] module-combine-sink.c: [alsa_output] sample rates too different, not adjusting (44100 vs. 33380).
> W: [pulseaudio] module-combine-sink.c: [alsa_output] sample rates too different, not adjusting (44100 vs. 28907).
> W: [pulseaudio] module-combine-sink.c: [alsa_output] sample rates too different, not adjusting (44100 vs. 31788).
> 
> From these logs, we can tell tunnel sink's sink-input is attached 
> first and its pop_cb function is called before alsa sink's sink-input 
> is attached. When alsa sink's sink-input is attached and its pop_cb 
> function is called, the tunnel sink's latency is still 0. It could be 
> because the gap between alsa sink attachment and tunnel sink attached 
> is too small and tunnel sink's buffer is still empty. The latency at 
> this time doesn't show the real latency that sink has. So this method 
> doesn't work. Actually this method only works when the other attached 
> sinks are already in stable playback status, meaning its latency is 
> constantly stable, before new sink is attached.

Even if the tunnel sink's buffer is "empty", its true latency isn't zero. So, pa_sink_input_get_latency_within_thread() returns incorrect values. I guess the latency information just isn't there immediately after the tunnel sink has become active (either created or unsuspended).
I'm not sure what would be the best way to address this - maybe
send_data() in module-tunnel.c should be modified to generate silence instead of calling pa_sink_render() until the latency information becomes available. The SINK_MESSAGE_UPDATE_LATENCY handler in module-tunnel.c seems like the right place to flag that now the latency information is available.

> For my previous questions about sink and sink-input. My intention is 
> to ask what is the difference between sink and sink-input. I saw both 
> of them are actually object instances which can register and accept 
> messages. After spending more time on reading the code, I seems to 
> know the difference now. Sink is actually a real/virtual sink module 
> instance which normally has a separate thread attached and can poll 
> and process async messages to it. While sink-input is corresponding 
> sink's representative at the other end of buffer. Is my understanding 
> correct?

The description for "sink" sounds quite right, but the "sink-input"
description isn't so accurate (or I just don't get what you mean).

I'll try an explanation: sink is an "output device". Sink input is a "playback stream" (so it's not a "representative" of the sink, it's a representative of whatever wants to send audio to the sink, in your case the sink inputs represent outputs of the combine sink). Any number of sink inputs can be connected to a sink. The sink mixes the inputs and sends the mixed audio to some backend. An alsa sink sends the audio to alsa, a tunnel sink sends the audio to another pulseaudio instance over the network and a combine sink sends the audio to multiple other sinks.

The clock in the system is provided by the sink: sink inputs can't just push data to a sink, it's always so that a sink pulls audio from the sink inputs when the backend needs more data.

The audio processing happens in a separate thread ("IO thread"), which is created by the sink. So, the audio processing code of sink inputs runs in the IO thread of the sink that the input is connected to. Each sink has has its own IO thread (with the exception of filter sinks, but that's not relevant here). In addition to the IO threads, there's the "main thread", i.e. the thread where pulseaudio's main() function runs, and where the system control is done.

The main thread communicates with the IO threads with messages, sent with pa_asyncmsgq_send() (synchronous) and pa_asyncmsgq_post() (asynchronous). In case of module-combine-sink, message passing is used also to communicate between the combine sink's IO thread and the IO threads of the sinks that the combine sink is connected to.

--
Tanu