RTSP stream, effect of max-lateness + qos

[Guillermo Rodriguez Garcia]

Hello,

El sáb, 13 may 2023 a las 11:23, Tim-Philipp Müller via gstreamer-devel (<
gstreamer-devel at lists.freedesktop.org>) escribió:

> Hi,
>
> However setting _both_ max-lateness=-1 and qos=false does work. WIth this
> I can see the video stream even if there is a very noticeable delay.
>
> I am trying to undestand how this works and I have some questions;
> hopefully someone can help.
>
> 1. Why setting max-lateness=-1 alone is not enough? If I understand
> correctly this should make the sink consider that no frames are late, so it
> should not be sending qos events. Why is it necessary to combine
> max-lateness=-1 with qos=false ?
>
> 2. What is the difference in practice between max-lateness=-1 + qos=false
> and sync=false? Both seem to achieve a similar effect, but I have read that
> "setting sync=false is almost never the right solution for timing problems".
>
>
> There are multiple places where frames can potentially be dropped here:
>
> a) at the videosink if "too late", and "max-lateness" controls what's
> considered "too late".
>
> b) at the videodecoder in response to QoS events from the sink (unless
> disabled via the "qos" property on the decoder or sink)
>
> c) at elements in between the videodecoder and the videosink, such as e.g.
> videoconvert/scale in response to QoS events from the sink (unless
> disabled via the "qos" property on the filter or sink) (you don't have
> those here of course)
>
> Qos events are always sent upstream, not only when a buffer is late (but
> elements won't drop anything then of course if the buffer was on time).
>
> If you just set *max-lateness=-1* on the sink you disable dropping of
> frames that are late, so the videosink will show every frame that arrives.
> However, it will then still send QoS events to upstream telling the
> decoder/filter how late that buffer was, and if it's quite a lot late then
> the decoder will try and skip some frames in order to catch up. You've just
> moved the frame dropping from the sink (plus decoder) to just the decoder.
> (I'm ignoring any filters in the middle that might exist in other
> pipelines, although they might also drop frames if they'd be way too late
> according to the last QoS event).
>
> If you just set *qos=false* no the sink, the decoder/filters will never
> skip frames in response to qos events, because no QoS events will be sent
> upstream. The sink will still drop frames that are "too late", because the
> max-lateness property is still set.
>

I see. This makes a lot of sense, thank you for the explanation.
I was assuming that the sink decides which frames are "too late" based on
max-lateness, so setting max-lateness=-1 would mean no frames are ever
considered late, and thus no qos events would be sent upstream. I see that
I was wrong.


>
> If you set both *qos=false* and *max-lateness=-1* then you disable all
> frame dropping, although then the decoder/sink will work slower than
> realtime. It's similar to just setting sync=false on the sink. so at some
> point there will be data before the decoder piling up that's not being
> processed in time, and at some point that will then cause other problems,
> e.g. if you receive via UDP the kernel-side UDP receive buffers will run
> full and you'll lose packets causing audio/video glitches. If you use TCP
> the sender might kick you out at some point if you're not reading data any
> more.
>

> If the performance is borderline you might get away with configuring the
> decoder to skip some things, e.g. *avdec_h264 skip-frame=1* would skip
> B-frames (assuming there are B-frames in the first place which might not be
> the case).
>
> The only way to avoid the 'backpressure' problem is to make sure things
> get dropped before or at the decoder, otherwise the decoder has to process
> everything and won't keep up and will fall more and more behind no matter
> what's happening after it.
>

Is there a way to tune the strategy for dropping frames at the decoder?
Ideally I would like to see the following
- Initially do not drop any "late" frames (otherwise basically I get a
frozen video which was my starting point)
- Only drop frames if the decoder itself cannot keep up, instead of based
on downstream qos events

?

Thanks,

Guillermo Rodriguez Garcia
guille.rodriguez at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20230515/bcead781/attachment.htm>