RTSP stream, effect of max-lateness + qos

[Tim-Philipp Müller]

Hi,

> However setting _both_ max-lateness=-1 and qos=false does work. WIth
> this I can see the video stream even if there is a very noticeable
> delay.
> 
> I am trying to undestand how this works and I have some questions;
> hopefully someone can help.
> 
> 1. Why setting max-lateness=-1 alone is not enough? If I understand
> correctly this should make the sink consider that no frames are late,
> so it should not be sending qos events. Why is it necessary to
> combine max-lateness=-1 with qos=false ?
> 
> 2. What is the difference in practice between max-lateness=-1 +
> qos=false and sync=false? Both seem to achieve a similar effect, but
> I have read that "setting sync=false is almost never the right
> solution for timing problems".

There are multiple places where frames can potentially be dropped here:

a) at the videosink if "too late", and "max-lateness" controls what's
considered "too late".

b) at the videodecoder in response to QoS events from the sink (unless
disabled via the "qos" property on the decoder or sink)

c) at elements in between the videodecoder and the videosink, such as
e.g. videoconvert/scale in response to QoS events from the sink (unless
disabled via the "qos" property on the filter or sink) (you don't have
those here of course)

Qos events are always sent upstream, not only when a buffer is late
(but elements won't drop anything then of course if the buffer was on
time).

If you just set max-lateness=-1 on the sink you disable dropping of
frames that are late, so the videosink will show every frame that
arrives. However, it will then still send QoS events to upstream
telling the decoder/filter how late that buffer was, and if it's quite
a lot late then the decoder will try and skip some frames in order to
catch up. You've just moved the frame dropping from the sink (plus
decoder) to just the decoder. (I'm ignoring any filters in the middle
that might exist in other pipelines, although they might also drop
frames if they'd be way too late according to the last QoS event).

If you just set qos=false no the sink, the decoder/filters will never
skip frames in response to qos events, because no QoS events will be
sent upstream. The sink will still drop frames that are "too late",
because the max-lateness property is still set.

If you set both qos=false and max-lateness=-1 then you disable all
frame dropping, although then the decoder/sink will work slower than
realtime. It's similar to just setting sync=false on the sink. so at
some point there will be data before the decoder piling up that's not
being processed in time, and at some point that will then cause other
problems, e.g. if you receive via UDP the kernel-side UDP receive
buffers will run full and you'll lose packets causing audio/video
glitches. If you use TCP the sender might kick you out at some point if
you're not reading data any more.

If the performance is borderline you might get away with configuring
the decoder to skip some things, e.g. avdec_h264 skip-frame=1 would
skip B-frames (assuming there are B-frames in the first place which
might not be the case).

The only way to avoid the 'backpressure' problem is to make sure things
get dropped before or at the decoder, otherwise the decoder has to
process everything and won't keep up and will fall more and more behind
no matter what's happening after it.

Cheers
 Tim

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20230513/a115b161/attachment-0001.htm>