[gst-devel] totem and osssink? (long)

[Benjamin Otte]

Ok, now for the comments. :)

Quoting Martin Soto <soto at informatik.uni-kl.de>:

> Properly programmed elements should be able to use whatever clock they
> receive. Even elements providing a clock should not count on being
> assigned their own clock. In case they are assigned a different clock,
> they should use it (and not their own) for synchronization. [I think
> not all sinks respect this rule. They should if we want to achieve real
> interoperation.]
> 
Every element not providing a clock obviously synchronizes to the given clock 
already. And I made alsasink and osssink sync to the given clock instead of 
their own, too. So there's esdsink left. Are there any more elements that 
supply their own clock yet?

> Element Time
> ------------
> 
> [...]
> This is achieved without touching the clock object assigned to the
> element. Elements contain a field, called base_time, that will be
> subtracted from the actual clock time in order to calculate the
> element time. gst_element_set_time just adjusts base_time properly to
> achieve this behavior.
> 
The element time is supposed to be identical to the timestamp of the buffer 
that is played at that time. So it should start with 0 whenever the element 
starts playback and be set to the timestamp received in a discont for that 
reason.
The core takes care of the basic functionality to do that (see GstElement's 
state changing function or the default handler for the discont event)

> Synchronization
> ---------------
>
My idea of how synchronization should work is this:
We provide a synchronization framework that needs to be explicitly requested 
by applications. This framework would take care of resynchronization of 
elements that are to be synchronized if that is possible.

This is actually quite a complex task because DISCONT events are not 
guaranteed to arrive at the same amount on video and audio sinks. DISCONTs 
just show a discontinuity in the stream. So if you stream a video where the 
video stream is broken and frames have to be skipped while the audio stream 
works fine (or the other way round) you end up with a lot of DISCONT events on 
one sink while there are none on the other sink.

I have some ideas about such a framework, so if anyone's interested, we can 
talk about it on IRC and/or start hacking on it.
 
> [As far as I understand it, it is not possible at all for two elements
> to synchronize if they don't receive a proper discont event. I thing
> most source elements don't send a discont at start, and that may be a
> cause for programs not working anymore after Benjamin's last changes.]
> 
Correct.
Not sending a discont at the start is the reason why releasing the audio 
device doesn't work. I'd encourage anyone hacking on elements to make sure 
that a DISCONT event is always sent first. (This is not that much of an issue 
for filters - they can just assume a discont arrives and they forward it, but 
it's important for srcs and demuxers.)

> [Is this true? I always use the convention that timestamps are
> associated to the start of the buffer, but I haven't seen it written
> anywhere.] The length of time the material should play is, on the
> other hand, rather determined by the characteristics of the stream
> 
Yes. The buffer's "timestamp" and "offset" parameter are associated with the 
beginning of the buffer, while the "duration" describes the time that elapses 
with the data of the buffer and the "offset_end" parameter describes the 
offset at the end of the buffer's data.
Keep in mind that some formats like raw audio require data to be in sequence 
(buffer1.timestamp + buffer1.duration == buffer2.timestamp) while others like 
subtitles do not. An keep in mind that some formats might require timestamps 
or offsets or durations or offset_ends while others might not.
This is unfortunately not documented. 
We will probably end up with different MIDI formats because of this.
And this is the reason why you can't just stop outputting sound in your DVD 
playing pipeline.

> The time base for the timestamps is usually arbitrary and determined
> by the media stream being played. In order for the sink elements to
> know how to properly interpret timestamps in a given media stream,
> their base time must be set based on the stream itself. For example,
> in order to play a video clip with a duration of 30 seconds, which is
> timestamped from 380s to 410s, the source element has to send a
> discont event with time 380s before sending the contents of the clip.
> That way, both the audio and video sinks will set their element times
> to 380s, and will start playing immediately as the first data buffers
> arrive.
> 
This is not quite right either. The timestamp of the first buffer inside a 
stream is 0. Unfortunately the definition of "first buffer inside a stream" is 
a bit fuzzy in some places - especially on infinite streams.

> Our current solution [which is actually a very clever hack from
> Benjamin, don't take me wrong here] works in sort of a "snap to grid"
> fashion. GstClock objects provide a gst_clock_get_event_time
> function. The value of gst_clock_get_event_time is usually identical
> to the value of gst_clock_get_time, i. e. it is the current clock
> time. However, if you invoke gst_clock_get_event_time twice in a short
> interval (how short is determined by the max-diff property in the
> clock object, whose default value is 2 seconds) you receive exactly
> the same value, namely, the time of the first invocation.
> 
I'd like to make sure that you all keep in mind that this is a hack and should 
be properly done in a synchronization framework because - as people already 
found out - it leads to all sorts of issues.
It's not a final solution, but it works well enough.

> It is very difficult to work around this problem in a satisfactory
> way. The only reliable solution I can think of would be identifying
> every discont event uniquely (with a serial number, for example), and
> having a separate event time in the clock for each discont. Of
> course, older event times can be discarded after some time, so you
> wouldn't have any issues with memory usage. Xine does something like
> this as well.
> 
The issue is still that DISCONT events don't need to be related to seeks as I 
outlined above. Of course we might still want to have the option to make 
DISCONT events related to seeks. I'm not sure what the best solution is here. 
We'll probably find out when designing the sync framework.

Benjamin