[gst-devel] information on gst_segment_clip() and the GstBaseSink Class

[Wim Taymans]

On Thu, 2006-02-23 at 14:22 +0530, Sameer Naik wrote:
> Hi Everybody,
> 	I was working on how avsync happens in gstreamer.
> Now, avsync is provided by the GstBaseSink Class...
> through my study of the basesink class i found out that avsync can only
> happen if a new segment is sent, otherwise ill get the message saying
> "Buffer Received without new segment: Cannot Sync to Clock"
Hi,

Synchronisation in the sinks is performed using the following
information:
 - base_time of the element
 - clock time
 - segment.start
 - buffer timestamp

The stream time in a sink is calculated as:

   clock_time - base_time

When setting a pipeline from READY to PLAYING or after a flushing seek,
the
clock is sampled and that value is set on _all_ elements as the
base_time. This makes sure that clock_time - base_time yields the same
value starting from 0 for
_all_ elements. This also means that stream_time is reset to 0 when
first starting a pipeline or after doing a flushing seek (that's why you
either need to send the flushing seek to the pipeline so it can reset
the stream_time or you reset the stream time yourself).

When a buffer arrives in the sink, first the clipping check is done:

  segment.start <= buffer timestamp < segment.stop

If the buffer timestamp is outside of the last received segment it is
dropped. The segment therefore defines the valid range of buffer
timestamps, this is usefull when doing accurate seeking (see later).

The equation buffer_timestamp - segment.start therefore yields a value
between 0 and the duration of the segment. If no segment arrived in the
sink, a segment.start value of 0 is assumed, which requires that buffer
timestamps start from 0 do do proper synchronisation.

Then the equation:

   (buffer_timestamp - segment.start) / segment.abs_rate + base_time

is used to compare the presentation time of the buffer against the
clock. Also the segment.rate is used to rescale the timestamps and speed
up/slow down playback.

A typical demuxer will produce buffers on the source pads with
timestamps that specify when they are to be displayed relative to
eachother. The segment.start and stop values define the valid timestamp
range and the 0 point of the timestamps. Many tricks can be done with
this system, like retimestamping by adjusting the segment values (see
gnonlin) or accurate seeking.

Suppose you have a demuxer and you do a seek to, let's say, time 11s.
Suppose there is a keyframe at time 10s the demuxer can send a
segment.start of 11s and can start sending the first keyframe buffer at
second 10. The decoder then decodes the keyframe but does not forward
the buffer (as it is outside of the segment range). When a buffer at 11s
is decoded, the decoder outputs that as the first buffer inside the
segment (the decoder can also output 10s, in which case the sink will
drop the buffer, although that only works for playback).

Since the 0 point of the buffer timestamps is defined by the
segment.start value, it is possible to output buffers from a demuxer
that don't start with a 0 timestamps. If this happens, the sink has to
receive the newsegment before it can properly sync on the buffer (and
that's why it gives this warning in the absence of a segment since it
cannot know if that was the intention...).

> 
> so heres by question:
> say my demultiplexer sends a new segment with the start time as the
> base_scr (in case of transport stream demux) and the end time as
> -1(infinity) ...and then starts sending video frames with a valid
> timestamp, after being decoded these frames will be sent to xvimagesink
> (which inherits the basesink)...
> in the sync'ing functionality of the basesink i found out that the start
> time and the end time of the received frame are extracted via the
> get_times function call and then gst_segment_clip is done to find out
> whether the frame shud be displayed or dropped...
> so i what i would like to know is when the segment clip is done are
> boundaries of the segment clipped too...
> for example..
> if the segment sent has starttime of 0 and end time of -1,
> next say a frame is sent with timestamp of 3...and say the frame has to
> be displayed and not dropped, so will the segment boundary be changed
> from

the segment boundary is only changed with a newsegment event or with a
flush (which resets all segment info), never with a buffer.

> [ 0 to -1 ] to
> [ 3 to -1 ]
> 
> what im confused on is that :say a frame with timestamp 3 is received
> and displayed and say the next frame with timestamp 2 arrives (not sure
> whether this could happen :), or case where the display time of the
> nextframe frame has passed )... how will it be dropped... (if the
> segment itself is clipped then,.. it makes sense to me, if not then i
> wonder how it happens..)

Out of order buffers are a mostly a decoder error and should be fixed,
there is no way a videosink can meaningfully display them without
buffering some random N frames. That said, the audiosink can handle out
of order buffers since it maintains a queue in a ringbuffer, but that is
just an implementation detail.

The frame should be dropped because it arrived too late in the sink but
that is currently not done. The frames will just be shown in the wrong
order. Alternatively since sinks also keep track of the last received
timestamp, they could drop any buffers before that, which is not done
either (but would be a trivial and acceptable patch for 0.10).

Hope that helps,
Wim

> 
> _________
> 
> next thing is that in the basesink theres this case in the switch
> statement
> Case GST_CLOCK_EARLY:
> which prints "Buffer to late, rendering anyway"
> ...what is meant by buffer to late....
> is it that the display time has passed, but its rendering the frame
> or is it that the display time has not yet come but it is rendering...

The return value is a bit weird, from the perspective of the clock, the
current timestamp is before it's current time (EARLY), which means that
the time has already passed in the clock. Currently the sink still
renders buffers that arived too late in the sink as fast as possible but
that's going to change eventually.

> 
> hope u guys understood my questions :)
> 
> plz reply
> thanks in advance
> 
> take care
> sameer
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
-- 
Wim Taymans <wim at fluendo.com>