Seek commands and tsdemux element

[Mario Montagud Climent]

Dear colleagues,

In a previous message, we discussed the performance and behavior of  
seek commands when using the tsdemux element.

We’ve run some tests when using both tsdemux and qtdemux elements, for  
MPEG2-TS and MP4 local files, respectively. These are the employed  
pipelines:

                                 +---------+   +------------+    
+---------+    +-------------+
                             |-> |  queue  |-> | h264_parser|-> |  
h264dec |->  | xvimagesink |
                                 +---------+   +------------+    
+---------+    +-------------+
+---------+    +---------+  |
| filesrc | -> | *demux  | -
+---------+    +---------+  |
                                 +---------+   +----------+     
+---------+    +---------------+
                             |-> | queue   |-> | aacparse |->  |    
faad  |->  | autoaudiosink |
                                 +---------+   +----------+     
+---------+    +---------------+

The seek command is as follows (target position = current position +  
5s, just for testing):

/* Seek to current_position + 5 seconds, just for testing */
static void send_seek_event (CustomData *data) {
   GstEvent *seek_event;

   /* We obtain the current position, needed for the seek command */
   if (!gst_element_query_position (data->pipeline, GST_FORMAT_TIME,  
&position)) {
     g_printerr ("Unable to retrieve current position.\n");
     return;
   }

   /* Seek event */
   seek_event = gst_event_new_seek (1.0, GST_FORMAT_TIME, GST_SEEK_FLAG_FLUSH ,
           GST_SEEK_TYPE_SET, position + 5000000000, GST_SEEK_TYPE_SET, -1);

   /* Send the event */
   gst_element_send_event (data->xvimagesink, seek_event);

}

When using tsdemux, there is a variable offset between the target seek  
position and the segment start field, varying between 2.34s and 2.42s  
for different seek operations (just for testing). However, when using  
qtdemux, there is a constant offset of 66.7ms (2 frames, 30 fps)  
between both values for different seek operations.

Are these values reasonable? Should the segment start field correspond  
with the exact target seek position?

The existence of these offset values can be seen as reasonable, as the  
decoder may need a key frame for starting the decoding process after  
the seek command, but the offset values are too large when using  
tsdemux (GoP length is 1s!) and constant when using qtdemux (2 frames)  
for different seek operations.

Our goal is to synchronize the playout of different, but related,  
streams (both on the same devices and across different devices) at the  
stream-level, based on relative and absolute timelines inserted into  
these streams (these values are obtained from the tsdemux element, we  
have slightly modified it).

We check the synchronization accuracy by comparing the timelines  
within the tsdemux elements (using pad probes), the stream time  
(calculated using the info from segments) and by using overlays (frame  
numbers) added during the encoding process, by using ffmpeg.

When using GoP lengths of 1s and 30fps, the synchronization accuracy  
between two mpeg2-ts streams is quite accurate, but we have also  
tested with other GoP lengths and, in these cases, it is necessary to  
add manual offsets for the seek commands to achieve an accurate  
synchronization. It is very strange and we do not understand why.

In these cases, several buffers are simultaneously pushed from the  
tsdemux to the decoder after the seek command and we lose the timeline  
references for synchronization. It does not seem to be any  
relationship between the number of pushed buffers and the offset  
between the target seek position and the segment start.
Is there any way to know how many (to-be-played) buffers are between  
the tsdemux element and the decoder?

We would appreciate it very much if you could clarify our doubts.

Thank you very much in advance!

Best regards,

Mario Montagud