Diagnosing Frame Sync Issues?

Tue May 14 00:37:24 PDT 2013

Hi,

On Mon, 2013-05-13 at 15:13 -0400, Stirling Westrup wrote:
> This may, or may not, be a GStreamer issue. I've written a video-wall
> application that splits a playing video into pieces and shows the
> pieces on zero-clients connected to the host computer via USB or
> Ethernet cable.
> 
> 
> This basic functionality is working, but when playing large (ie 4K
> resolution) videos, we sometimes see momentary differences between
> when different monitors get updated with the next frame of video.
> 
> 
> Now, I can easily imagine a number of places that a video frame may be
> delayed so as to show after its neighbor:
> 
> 
> 1) Inside the GStreamer pipeline.
> 
> 2) Inside the network stack
> 
> 3) due to USB bandwidth issues.
> 
> 
> So far the only code in GStreamer that tries to maintain sync is that
> in the queue that handles each output monitor. The assumption made was
> that that would be sufficient. I could (with a lot of rewriting)
> replace all N output queues with a single N-way multiqueue, but I'm
> unsure that such a move is actually necessary.

  That's not quite the reality.
  1) The queue's job is to dedicate a thread to the rendering process
(i.e. the thread;s sole taks is to pick the next buffer available from
the queue (if any) and push it to the sink). Switching to multiqueue
won't change anything.
  2) The synchronization (i.e. waiting for the render time) is done in
the sink.

  Based on what you mentioned, you have N different sinks for the
various outputs. I'll go under that assumption.

  The issue is that the delay between the moment GStreamer (and the
GstBaseSink base class) says "yes, this buffer time corresponds to this
clock time, render it at this clock time" and the moment it actually
gets displayed to the user is:
  1) not immediate
  2) not the same for every sink

  In order to get perfect sync between your displays (and also with the
audio, which you don't mention), the sinks need to know the latency
introduced in the render process.

  The current synchronization model in video sinks (and all other sink
element that use the standard GstBaseSink synchronization model) is
"wait-then-render".
  When a buffer comes in, GstBaseSink will figure out a target clock
running time, wait for that moment and then call your sink's "render"
method (which does the actual display/output/...).

  And this is where it gets tricky...

  In order to compensate for that render delay in the wait-then-render
model, your sink needs to report that to the base class (render_delay
property), so that it will subtract that value from the target clock
running_time and call your render method that much earlier...
  ... which will only work if:
   1) you can efficiently calculate that delay
   2) it is constant (some jitter might be acceptable)

  If you can't calculate that delay and if it's not roughly constant...
you're out of luck with the current model.
  You could try to do some empirical testing and come up with some
"better" values ("in average it's 100ms over usb and 50ms over
ethernet") but it will only mitigate the problem, it won't solve it.
This might be acceptable though for your use-case, you might want to try
that.

  If you want to get "perfect" synchronization, you need to:
  1) Have systems which can report to you at what time a certain frame
was displayed (so you can internally calculate the remote system latency
and display rate)
  2) Have systems which can ask you for the frame to be displayed at a
certain target remote time.
  2) And switch to a synchronization model which is more like how audio
sinks work in Gstreamer (they are pull-based with an internal
ring-buffer and decide which frame to give the audio subsystem at a
given time).

  Note that Barco does such multi-screen displays (for control centers
and so forth), but they have solved the problem by having "smart"
displays which do the decoding/synchronization/display. The data is fed
to the screens via RTP and all displays have got a synchronized clock.
Doing it that way essentially moves the whole latency/synchronization
issue to the displays.
  More info in this talk :
http://gstconf.ubicast.tv/videos/displaying-sychronized-video-2/

  Hope this helps,

  Edward

P.S. Note that "perfect" is very subjective especially when it comes to
audio/video synchronization. Even if you have you audio/video
synchronized in output ... if your speakers are 5meters away from where
you sit... the sound will arrive to your ears 16ms late ... Joy :)

> 
> 
> Does anyone have any suggestions of ways, techniques, whatever to try
> to narrow down cause of the frame sync issue before I go rewriting
> great chunks of code. I would hate to do all that just to find out it
> was a USB driver issue all along...
> 
> 
> -- 
> Stirling Westrup
> Programmer, Entrepreneur.
> https://www.linkedin.com/e/fpf/77228
> http://www.linkedin.com/in/swestrup
> http://technaut.livejournal.com
> http://sourceforge.net/users/stirlingwestrup
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel