[gst-devel] Clocking fix

[Martin Soto]

On Tue, 2004-03-23 at 19:05, Benjamin Otte wrote: 
> I still consider "event time" a broken concept. Sync should be done
> explicitly, not implicitly by "snap to grid".

I agree with you. But for the time being we'd rather keep things
running. My recent scheduler work has brought me some additional ideas
on how to do sync The Right Way (TM), but more on that later.

> Apart from that I don't really get why this is needed.
> 
> Could you give an example where it helps?

Right. It is not that easy to explain (and of course I was lazy and
hoped that you would get it... :-) but I'll try. Think of an audio sink
that is able to buffer up to 0.5s of sound material in the sound card's
hardware buffer. Let's say we send this sink two chunks of audio, both
of them 10s long, the first one timestamped from 20s to 30s, and the
second one timestamped from 40s to 50s. Let's also say that the clock
starts in clock time (ct) equal 0s.

So, the sink receives a discont at the start of the first chunk,
containing time 20s. It does a gst_element_set_time that makes the
association element time (et): 20s <==> ct: 0s. Sound material follows
the discont directly and it plays instantly, since the card's buffer is
still empty. So far, so good.

Now, we approach the end of the first chunk. The whole audio material
has been received by the sink, and the discont for the second chunk
(containing time 40s) arrives. The question is, at what time does the
sink element see it?

Actually, as soon as the card's playback buffer is full, you have a
delay of 0.5s between the time the sink element receives a buffer, and
the time the material in that buffer starts playing. That means that the
second discont event arrives to the sink 0.5s before the time the second
chunk should start playing, that is, at element time 29.5s (clock time
9.5s). The element sees the discont and sets its time to 40s, which will
build the association et: 40s <==> ct: 9.5s.

Now, the first buffer of the second chunk arrives, with timestamp 40s.
The element time is already at 40s, which means the material should play
right now. However, we still have a backlog of 0.5s audio material in
the card's buffer. The sink will end up dropping material in order to
"resynchronize", but there's no actual lack of synchrony, it is just the
discont wasn't handled properly. This, of course, causes a lot of
playback "roughness" around disconts.

As you can see, the problem happens even with a single element. I
haven't even mentioned event time and the snap-to-grid thingie so far.
But they come now. My first idea to solve the problem, was to change the
gst_element_set_time call in the sink to something like

  gst_element_set_time (element, time_from_discont - current_delay);

That would work for a single element, but actually breaks the sync.
Image now that you have a video sink playing together with essentially
no hardware delay. The sequence of events around the discont looks like
this:

  1. Audio sink receives discont at time et: 29.5s (ct: 9.5s)
  2. Since delay is 0.5s and discont time is 40s, the audio sink sets
element time to et: 39.5s <==> ct: 9.5s
  3. Clock's event time is set to 9.5s.
  4. Video sink receives discont at time et: 10s
  5. Since video sink has no delay, it runs gst_element_set_time (40s).

Here comes the problem. gst_element_set_time associates its parameter
with the current clock's event time, which is still 9.5s. So you get the
association 40s <==> 9.5s, which results in a lack of synchrony of 0.5s.

Now, if you look at my routines, they avoid this problem by always
associating event time with the time in the event (which is guaranteed
to be the same for both elements), but by setting the current element
time based on the delay. So, in step two, the audio element calls
gst_element_set_time_delay, which sets element time to 39.5s, but sets
event time to 10s. This way everything works again.

As I said, I think also that we need a better solution, but for the
moment, this will do. I'd rather commit my patch know since it doesn't
break anything, and depend on it until we have a suitable infrastructure
in place.

Cheers (und viel Spaß dabei ;-)

M. S.
-- 
Martin Soto <soto at informatik.uni-kl.de>
Universität Kaiserslautern