[pulseaudio-discuss] a question about audio synchronization between local sink and tunnel sink

[Sun, Xiaodong]

Tanu,

I have modified code like this in module-combine-sink.c after exchanging idea with you.

In function sink_input_pop_cb():  add these lines before calling request_memblock()
    ...

    if (o->just_attached) {
        pa_usec_t sink_input_latency, sink_latency, total_latency1, total_latency2;
        struct output *j;

        sink_input_latency = pa_sink_input_get_latency_within_thread(i, &sink_latency);
        sink_input_latency += sink_latency;
        total_latency1 = sink_input_latency + pa_bytes_to_usec(pa_memblockq_get_length(o->memblockq), &o->sink_input->sample_spec);
        pa_log_warn("[%s] latency %0.2fms, output buffer latency %0.2fms.", o->sink_input->sink->name, (double)total_latency1/PA_USEC_PER_MSEC, (double)(total_latency1-sink_input_latency)/PA_USEC_PER_MSEC);

        /* OK, let's send this data to the other threads */
        PA_LLIST_FOREACH(j, o->userdata->thread_info.active_outputs) {
            if (j != o) {
                sink_input_latency = pa_sink_input_get_latency_within_thread(j->sink_input, &sink_latency);
                sink_input_latency += sink_latency;
                total_latency2 = sink_input_latency + pa_bytes_to_usec(pa_memblockq_get_length(j->memblockq), &j->sink_input->sample_spec);
                pa_log_warn("Peer: [%s] latency %0.2fms, output buffer latency %0.2fms.", j->sink_input->sink->name, (double)total_latency2/PA_USEC_PER_MSEC, (double)(total_latency2-sink_input_latency)/PA_USEC_PER_MSEC);
                if (total_latency1 > total_latency2)
                    pa_memblockq_drop(o->memblockq, pa_usec_to_bytes(total_latency1-total_latency2, &o->sink_input->sample_spec));
                else if (total_latency1 < total_latency2)
                    pa_memblockq_rewind(o->memblockq, pa_usec_to_bytes(total_latency2-total_latency1, &o->sink_input->sample_spec));
                break;
            }
        }

        o->just_attached = 0;
    }

In function sink_input_attach_cb(): add  o->just_attached = 1;

In function sink_input_detach_cb(): add  o->just_attached = 0;


Then I got following logs when playing audio. Of course un-synchronization issue is still there.

W: [module-tunnel] module-combine-sink.c: [tunnel_sink] attached
W: [module-tunnel] module-combine-sink.c: [tunnel_sink] latency 0.00ms, output buffer latency 0.00ms.
W: [alsa-sink] module-combine-sink.c: [alsa_output] attached
W: [alsa-sink] module-combine-sink.c: [alsa_output] latency 0.88ms, output buffer latency 0.00ms.
W: [alsa-sink] module-combine-sink.c: Peer: [tunnel_sink] latency 0.00ms, output buffer latency 0.00ms.
W: [pulseaudio] module-combine-sink.c: [alsa_output] sample rates too different, not adjusting (44100 vs. 33380).
W: [pulseaudio] module-combine-sink.c: [alsa_output] sample rates too different, not adjusting (44100 vs. 28907).
W: [pulseaudio] module-combine-sink.c: [alsa_output] sample rates too different, not adjusting (44100 vs. 31788).

From these logs, we can tell tunnel sink's sink-input is attached first and its pop_cb function is called before alsa sink's sink-input is attached. When alsa sink's sink-input is attached and its pop_cb function is called, the tunnel sink's latency is still 0. It could be because the gap between alsa sink attachment and tunnel sink attached is too small and tunnel sink's buffer is still empty. The latency at this time doesn't show the real latency that sink has. So this method doesn't work. Actually this method only works when the other attached sinks are already in stable playback status, meaning its latency is constantly stable, before new sink is attached.


For my previous questions about sink and sink-input. My intention is to ask what is the difference between sink and sink-input. I saw both of them are actually object instances which can register and accept messages. After spending more time on reading the code, I seems to know the difference now. Sink is actually a real/virtual sink module instance which normally has a separate thread attached and can poll and process async messages to it. While sink-input is corresponding sink's representative at the other end of buffer. Is my understanding correct?

Thanks! 

- Xiaodong

-----Original Message-----
From: Tanu Kaskinen [mailto:tanuk at iki.fi] 
Sent: Thursday, October 25, 2012 1:39 AM
To: Sun, Xiaodong
Cc: pulseaudio-discuss at lists.freedesktop.org
Subject: Re: [pulseaudio-discuss] a question about audio synchronization between local sink and tunnel sink

On Wed, 2012-10-24 at 23:49 +0000, Sun, Xiaodong wrote:
> Hi, Tanu,
> 
> First of all, thanks for your suggestion. This is definitely the 
> simplest way to solve this issue as long as it works as you expect, as 
> adding a new API in PA core also means changing application is 
> required.
> 
> I added following lines right before request_memblock(o, nbytes) in 
> module-combine-sink.c function sink_input_pop_cb().
> 
>     pa_usec_t sink_input_latency, sink_latency, total_latency;
>     sink_input_latency = pa_sink_input_get_latency_within_thread(i, &sink_latency);
>     total_latency = sink_input_latency + sink_latency;
>     pa_memblockq_seek(o->memblockq, pa_usec_to_bytes(total_latency, 
> &o->sink->sample_spec), PA_SEEK_RELATIVE, TRUE);
> 
> Then I define pa_sink_input_get_latency_within_thread() function in 
> sink_input.c
> 
> But the new code gave me an error log message like this when I am 
> trying to create a combine sink (combining a local alsa sink and a 
> tunnel sink):
> 
> E: [alsa-sink] memblockq.c: Assertion 'length % bq->base == 0' failed at pulsecore/memblockq.c:613, function pa_memblockq_drop(). Aborting.
> Aborted

This might be caused by using o->sink->sample_spec in the
pa_memblockq_seek() call. You should use o->sink_input->sample_spec when dealing with the output's memblockq. o->sink is the sink that the output is connected to, and it may have a different sample spec than
o->sink_input, which uses the sample spec of the combine sink. The
output's memblockq contains data in the format of the combine sink.

> Here is my questions:
> 
> 1. Is that seek function call correct?

What the call does is that it moves the write index of the memblockq forward. That will cause a gap (silence) in the memblockq, which is not what you want, especially not every time the sink_input_pop_cb() is called. You should seek in the buffer only the first time the function is called. By "seek" I don't necessarily mean pa_memblockq_seek() - rather than moving the write index, you'll need to move the read index.
That's done with pa_memblockq_drop() or pa_memblock_rewind(), depending on whether you need to move the read index forward or backward.

It's not clear to me what the exact algorithm should be for figuring out how much to move the read index. I think you need to somehow keep track of how much in total has been played (meaning how many bytes have come out of the speakers), I'll call that number the "played_count".

When the first output is created, nothing has come out of the speakers.
When sink_input_pop_cb is called for the first time ever, the played_count is actually negative, because there is some latency. The first output doesn't have to do any seeking, since there are no other outputs to synchronize with.

When the second output is created, sink_input_pop_cb will somehow have to figure out what the current played_count is, and if the first chunk in the output's memblockq isn't the first chunk that has ever been gone through the combine sink, the output needs to know how much data has there been before the first chunk in the memblockq. I'll call that number the "skipped_count".

Once you have somehow figured out the current played_count and the skipped_count, and you have the total_latency of the second output too, you can calculate how much and to what direction the read index of the output's memblockq needs to be moved (negative result means moving backward and positive forward):

    move_amount = played_count - skipped_count + total_latency;

> 2. Following is my understanding of your suggestion, is it true?
>     The new output stream need to seek to a starting point represented 
> using latencies (should include its own latency and another output 
> stream latency). In my case, there are two output streams (alsa sink 
> output and tunnel sink output). Each of them have a separate buffer 
> (memblockq ?). But the data of these buffers comes from the same 
> virtual sink (called combined). When new output stream has smaller 
> latency (latency_1) than the other one (latency_2), then the new 
> output stream need to skip to starting point (latency_2 - latency_1) 
> when filling its buffer with data from virtual sink. However when new 
> output stream has bigger latency than the other one, then the new 
> output stream need to fill silence data with size (latency_1 - 
> latency_2)*sample_rate*sample_size into its buffer to compensate this 
> delay. If my understanding is correct, then the above modification is 
> not enough.

Sounds roughly like what I described above. Note that the silence generation happens automatically. At the beginning the memblockq read index is 0. If you then move the read index backward (pa_memblockq_rewind()), the read index becomes negative. The first data chunk in the buffer starts at index 0, so when data is read from the memblockq for the first time with pa_memblockq_peek(),
pa_memblockq_peek() will generate silence.

> 3. What is the purpose of sink_input? Why we can not use sink 
> directly?

Sinks get their data from sink inputs. I don't know what you mean by "using sink directly".

--
Tanu