Diagnosing Frame Sync Issues?

Stirling Westrup swestrup at gmail.com
Tue May 14 09:01:23 PDT 2013


On Tue, May 14, 2013 at 3:37 AM, Edward Hervey <bilboed at gmail.com> wrote:

Thanks hugely for your response. I very much appreciate the time you took
to explain the inner model of the sync process.


>
> On Mon, 2013-05-13 at 15:13 -0400, Stirling Westrup wrote:
> > This may, or may not, be a GStreamer issue. I've written a video-wall
> > application that splits a playing video into pieces and shows the
> > pieces on zero-clients connected to the host computer via USB or
> > Ethernet cable.
> >
> >
> > This basic functionality is working, but when playing large (ie 4K
> > resolution) videos, we sometimes see momentary differences between
> > when different monitors get updated with the next frame of video.
> >
> >
> > Now, I can easily imagine a number of places that a video frame may be
> > delayed so as to show after its neighbor:
> >
> >
> > 1) Inside the GStreamer pipeline.
> >
> > 2) Inside the network stack
> >
> > 3) due to USB bandwidth issues.
> >
> >
> > So far the only code in GStreamer that tries to maintain sync is that
> > in the queue that handles each output monitor. The assumption made was
> > that that would be sufficient. I could (with a lot of rewriting)
> > replace all N output queues with a single N-way multiqueue, but I'm
> > unsure that such a move is actually necessary.
>
>   That's not quite the reality.
>   1) The queue's job is to dedicate a thread to the rendering process
> (i.e. the thread;s sole taks is to pick the next buffer available from
> the queue (if any) and push it to the sink). Switching to multiqueue
> won't change anything.
>   2) The synchronization (i.e. waiting for the render time) is done in
> the sink.
>

Ah. Thanks. I had not really understood that, despite reading both the main
manual and the element writing manual.


>
>   Based on what you mentioned, you have N different sinks for the
> various outputs. I'll go under that assumption.
>

Correct.


>
>   The issue is that the delay between the moment GStreamer (and the
> GstBaseSink base class) says "yes, this buffer time corresponds to this
> clock time, render it at this clock time" and the moment it actually
> gets displayed to the user is:
>   1) not immediate
>   2) not the same for every sink
>
>   In order to get perfect sync between your displays (and also with the
> audio, which you don't mention), the sinks need to know the latency
> introduced in the render process.
>
>   The current synchronization model in video sinks (and all other sink
> element that use the standard GstBaseSink synchronization model) is
> "wait-then-render".
>   When a buffer comes in, GstBaseSink will figure out a target clock
> running time, wait for that moment and then call your sink's "render"
> method (which does the actual display/output/...).
>
>   And this is where it gets tricky...
>
>   In order to compensate for that render delay in the wait-then-render
> model, your sink needs to report that to the base class (render_delay
> property), so that it will subtract that value from the target clock
> running_time and call your render method that much earlier...
>   ... which will only work if:
>    1) you can efficiently calculate that delay
>    2) it is constant (some jitter might be acceptable)
>
>   If you can't calculate that delay and if it's not roughly constant...
> you're out of luck with the current model.
>   You could try to do some empirical testing and come up with some
> "better" values ("in average it's 100ms over usb and 50ms over
> ethernet") but it will only mitigate the problem, it won't solve it.
> This might be acceptable though for your use-case, you might want to try
> that.
>

This sounds like a likely approach. I've been looking around at our
competition and the current 'default' synchronization that GStreamer
achieves is already better than many commercial products, so any
improvement we can achieve would be good. We don't need 'perfect'. (And
you're right, it probably can't be achieved with this hardware, but we are
going for 'good, cheap, video-wall' so ...)

We haven't much worried about sound synchronization because many clients
won't have sound turned on, and those that do will have systems that
introduce delays (like surround sound systems) and so we'll probably need a
hand-tuning parameter for sound sync in any particular space.

I will try adding some extra time to the 'render-delay' parameter in
xvimagesink and see what I can accomplish, but that does make me curious.
What is the downside to being over-aggressive in setting that parameter.
ie, if I set it to 3 seconds, what is the downside, other than delaying
start of video playing by 3 seconds?

I ask because we can't distinguish if the client has connected their
zero-clients via USB or via Ethernet, and so can't give different values
for delay that automatically compensate. Thus we're likely to set a default
worst-case delay and let the client tune it with some sort of config
parameter.


_______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>



-- 
Stirling Westrup
Programmer, Entrepreneur.
https://www.linkedin.com/e/fpf/77228
http://www.linkedin.com/in/swestrup
http://technaut.livejournal.com
http://sourceforge.net/users/stirlingwestrup
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20130514/82aadf5a/attachment-0001.html>


More information about the gstreamer-devel mailing list