Two glvideomixers on a tee freeze on input caps change (Was: Re: How does a muxer deal with packet loss?)

[Michiel Konstapel]

> If the muxers aren't the culprit, could it be the mixers? Those are 
> aggregators, too. How do those deal with missing or late inputs?

I found something! Not sure what it means, though. I found I could 
trigger the freeze (or at least, something that looks the same) by 
changing the window size of a screensharing WebRTC video track.

When the video went into both the preview and main glvideomixers, the 
pipeline locked up. Feeding it into only one of the mixers (didn't 
matter which one) it continued just fine. In the locked up state, one of 
the mixers' sinkpads/proxypads would disagree on the caps.

It looks like the same thing happens, or can happen, when switching 
between the real source (1280x720, variable framerate?) and the 
imagefreeze source (1920x1080 at 25fps). At least, I think that's what I'm 
seeing in the attached screenshot of the dump.

Why would that happen? I found this:

https://gstreamer-bugs.narkive.com/hcKdsAA0/bug-780682-new-compositor-corrupts-output-when-input-caps-change-while-running#post17

"videoaggregator: delay using new caps from a sink pad until the next 
buffer in the queue is taken

When caps changes while streaming, the new caps was getting processed
immediately in videoaggregator, but the next buffer in the queue that
corresponds to this new caps was not necessarily being used immediately,
which resulted sometimes in using an old buffer with new caps. Of course
there used to be a separate buffer_vinfo for mapping the buffer with its
own caps, but in compositor the GstVideoConverter was still using wrong
info and resulted in invalid reads and corrupt output.

This approach here is more safe. We delay using the new caps
until we actually select the next buffer in the queue for use.
This way we also eliminate the need for buffer_vinfo, since the
pad->info is always in sync with the format of the selected buffer."

Which sounds related, but could also be a completely innocent bug fix.

Anyone have an idea why the mixers wouldn't just carry on with the new 
input geometry? Could the variable frame rate from webrtcbin be an 
issue? I don't know why it's 0/1, it's supposed to be a 25fps stream as 
well.

Kind regards,
Michiel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: caps.png
Type: image/png
Size: 247164 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210623/b59dffd3/attachment-0001.png>