[gst-devel] information on gst_segment_clip() and the GstBaseSink Class

[Sameer Naik]

Thanks All,
	that was loads of info, really sored things out for me.
i have decided to patch the gstbasesink, however will take some time
since i have to first understand the whole arcietecture that goes into
it and at the same time make sure that i dont introduce some code that
would result in inconsistencies in the current code..

shall inform u guys when i get cracking on this, and also would need ur
help in accomplishing this task. 

thanks a bunch for all that help provided.
take care
Sameer

On Thu, 2006-02-23 at 14:38 +0100, Wim Taymans wrote:
> On Thu, 2006-02-23 at 14:22 +0530, Sameer Naik wrote:
> > Hi Everybody,
> > 	I was working on how avsync happens in gstreamer.
> > Now, avsync is provided by the GstBaseSink Class...
> > through my study of the basesink class i found out that avsync can only
> > happen if a new segment is sent, otherwise ill get the message saying
> > "Buffer Received without new segment: Cannot Sync to Clock"
> Hi,
> 
> Synchronisation in the sinks is performed using the following
> information:
>  - base_time of the element
>  - clock time
>  - segment.start
>  - buffer timestamp
> 
> The stream time in a sink is calculated as:
> 
>    clock_time - base_time
> 
> When setting a pipeline from READY to PLAYING or after a flushing seek,
> the
> clock is sampled and that value is set on _all_ elements as the
> base_time. This makes sure that clock_time - base_time yields the same
> value starting from 0 for
> _all_ elements. This also means that stream_time is reset to 0 when
> first starting a pipeline or after doing a flushing seek (that's why you
> either need to send the flushing seek to the pipeline so it can reset
> the stream_time or you reset the stream time yourself).
> 
> When a buffer arrives in the sink, first the clipping check is done:
> 
>   segment.start <= buffer timestamp < segment.stop
> 
> If the buffer timestamp is outside of the last received segment it is
> dropped. The segment therefore defines the valid range of buffer
> timestamps, this is usefull when doing accurate seeking (see later).
> 
> The equation buffer_timestamp - segment.start therefore yields a value
> between 0 and the duration of the segment. If no segment arrived in the
> sink, a segment.start value of 0 is assumed, which requires that buffer
> timestamps start from 0 do do proper synchronisation.
> 
> Then the equation:
> 
>    (buffer_timestamp - segment.start) / segment.abs_rate + base_time
> 
> is used to compare the presentation time of the buffer against the
> clock. Also the segment.rate is used to rescale the timestamps and speed
> up/slow down playback.
> 
> A typical demuxer will produce buffers on the source pads with
> timestamps that specify when they are to be displayed relative to
> eachother. The segment.start and stop values define the valid timestamp
> range and the 0 point of the timestamps. Many tricks can be done with
> this system, like retimestamping by adjusting the segment values (see
> gnonlin) or accurate seeking.
> 
> Suppose you have a demuxer and you do a seek to, let's say, time 11s.
> Suppose there is a keyframe at time 10s the demuxer can send a
> segment.start of 11s and can start sending the first keyframe buffer at
> second 10. The decoder then decodes the keyframe but does not forward
> the buffer (as it is outside of the segment range). When a buffer at 11s
> is decoded, the decoder outputs that as the first buffer inside the
> segment (the decoder can also output 10s, in which case the sink will
> drop the buffer, although that only works for playback).
> 
> Since the 0 point of the buffer timestamps is defined by the
> segment.start value, it is possible to output buffers from a demuxer
> that don't start with a 0 timestamps. If this happens, the sink has to
> receive the newsegment before it can properly sync on the buffer (and
> that's why it gives this warning in the absence of a segment since it
> cannot know if that was the intention...).
> 
> > 
> > so heres by question:
> > say my demultiplexer sends a new segment with the start time as the
> > base_scr (in case of transport stream demux) and the end time as
> > -1(infinity) ...and then starts sending video frames with a valid
> > timestamp, after being decoded these frames will be sent to xvimagesink
> > (which inherits the basesink)...
> > in the sync'ing functionality of the basesink i found out that the start
> > time and the end time of the received frame are extracted via the
> > get_times function call and then gst_segment_clip is done to find out
> > whether the frame shud be displayed or dropped...
> > so i what i would like to know is when the segment clip is done are
> > boundaries of the segment clipped too...
> > for example..
> > if the segment sent has starttime of 0 and end time of -1,
> > next say a frame is sent with timestamp of 3...and say the frame has to
> > be displayed and not dropped, so will the segment boundary be changed
> > from
> 
> the segment boundary is only changed with a newsegment event or with a
> flush (which resets all segment info), never with a buffer.
> 
> > [ 0 to -1 ] to
> > [ 3 to -1 ]
> > 
> > what im confused on is that :say a frame with timestamp 3 is received
> > and displayed and say the next frame with timestamp 2 arrives (not sure
> > whether this could happen :), or case where the display time of the
> > nextframe frame has passed )... how will it be dropped... (if the
> > segment itself is clipped then,.. it makes sense to me, if not then i
> > wonder how it happens..)
> 
> Out of order buffers are a mostly a decoder error and should be fixed,
> there is no way a videosink can meaningfully display them without
> buffering some random N frames. That said, the audiosink can handle out
> of order buffers since it maintains a queue in a ringbuffer, but that is
> just an implementation detail.
> 
> The frame should be dropped because it arrived too late in the sink but
> that is currently not done. The frames will just be shown in the wrong
> order. Alternatively since sinks also keep track of the last received
> timestamp, they could drop any buffers before that, which is not done
> either (but would be a trivial and acceptable patch for 0.10).
> 
> Hope that helps,
> Wim
> 
> > 
> > _________
> > 
> > next thing is that in the basesink theres this case in the switch
> > statement
> > Case GST_CLOCK_EARLY:
> > which prints "Buffer to late, rendering anyway"
> > ...what is meant by buffer to late....
> > is it that the display time has passed, but its rendering the frame
> > or is it that the display time has not yet come but it is rendering...
> 
> The return value is a bit weird, from the perspective of the clock, the
> current timestamp is before it's current time (EARLY), which means that
> the time has already passed in the clock. Currently the sink still
> renders buffers that arived too late in the sink as fast as possible but
> that's going to change eventually.
> 
> > 
> > hope u guys understood my questions :)
> > 
> > plz reply
> > thanks in advance
> > 
> > take care
> > sameer
> > 
> > 
> > 
> > -------------------------------------------------------
> > This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> > that extends applications into web and mobile media. Attend the live webcast
> > and join the prime developer group breaking into this new coding territory!
> > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> > _______________________________________________
> > gstreamer-devel mailing list
> > gstreamer-devel at lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/gstreamer-devel