Problem of sync between multiple pipeline and meaning of PTS and DTS

[Nicolas Dufresne]

Le jeudi 26 août 2021 à 18:45 +0900, 이병헌 via gstreamer-devel a écrit :
> Hi, I’m new on gstreamer now
> And my first language is not an English
> So if my questions are not good to be understood I would say sorry first..
> 
> Anyway now I’m trying to make application that need function of synchronize
> buffers between multiple pipeline
> So I’m trying to sync those buffers by Lockstep method with their timestamp
> value in those buffers
> But there have three problem
> 
> This is my pipeline blueprint
> 
> rtspsrc ! h265depay ! av_dech265 ! videoscale ! videoconvert ! caps-filter !
> queue ! videoscale ! tensor_converter ! tensor_filter ! tensor_sink —— get
> buffer from sink by callback and put buffers to — > Ring buffer - - \
> ! rtspsrc ! h265depay ! av_dech265 ! videoscale ! videoconvert ! caps-filter !
> queue ! videoscale ! tensor_converter ! tensor_filter ! tensor_sink —— get
> buffer from sink by callback and put buffers to — > Ring buffer —- - >
> lockstep sync —> some association logics ...
> ! rtspsrc ! h265depayl ! av_dech265 ! videoscale ! videoconvert ! caps-filter
> ! queue ! videoscale ! tensor_converter ! tensor_filter ! tensor_sink —— get
> buffer from sink by callback and put buffers to — > Ring buffer —/
>  
> First , PTS is not wall clock time or ntp time
> Second, DTS value is always -1
> Third, when I check pts values at source element, pts values are always
> increment. But when I check my ring buffer element, some value of pts are
> decrease between adjacent buffer, this mean order of input buffer can be
> shuffled while streaming
> 
> So.. my questions are these
> 
> 1. How to get wall clock PTS value from rtsp packets in gstreamer?

In GStreamer we have the concept of "running-time", this is the timescale you
must use to compare GST_BUFFER_DTS_OR_PTS() (that macro takes care of picking
the right TS) against each other.

In order to get running-time from a buffer PTS, you have to track the GstSegment
that is propagated using the GST_EVENT_SEGMENT. You can then convert using
gst_segment_to_running_time() or gst_segment_to_running_time_full(). The full
version supports negative running time, this might not be needed for your use
case.

> 2. Meaning of DTS value -1

DTS stands for decoting timestamp. In your example, you are dealing with raw
data, so decoding does not make sense, so that TS is set to
GST_CLOCK_TIME_INVALID. It's not really -1, since this is unsigned integer.

Note that DTS is optional, it is mostly useful for streams were the decoding
order differ from the presentation order, or when there is a delay (like in
opus) with some initial data that should be skipped.

> 3. Is it possible to get different order of buffer from input buffer order at
> some elements in pipeline? If can, then do I have to sort those buffers with
> their timestamp with some priority queue rather than ring buffer ?

In your case, it looks like getting backward timestamp is a bug. Could be on the
RTSP server side, this requires deep investigation to figure-out.

> 
> Sorry for writing that hard to understand 
> and I would give more inform if it is not enough to answer
> 
> Thanks