[videocrop/compositor] Change output of videocrop just in time using GstVideoCropMeta metadata

[Arjen Veenhuizen]

My goal is to create a composited image with a low resolution/low quality
full frame and one or multiple high quality cut-outs, neatly aligned and
scaled to properly fit the low resolution full frame. (See [1] for an
impression)

I have roughly the following pipeline wrapped in python code:



I use videocrop to crop a specific region per frame. This region (x, y,
width, height), however, also changes for each frame. I cannot rely on out
of band pad-probes and change the videocrop element properties (left, right,
top, bottom) in a pad probe callback due to overhead so I have (and want to)
do this in-band. Therefore I have modified the videocrop element to accept
GstVideoCropMeta metadata and use that to define the output per frame. The
videocrop element will append GstvideoCropMeta metadata to its output buffer
as well instructing the modified compositor element on how to position and
scale its input frame on its sinkpad. 

Problem 1: 
If I understand things correctly, I am already too late to set the output
frame width/height when I am parsing the input buffer in the videocrop
element. If I would still output a frame with the required width/height (as
signalled in the GstVideoCropMeta metadata), I would potentially start
writing in unallocated memory (because my output frame is simply not big
enough) causing visual distortions and sooner or later a segfault. As a
partial work-around, I signal the width and height of the next output frame
one frame in advance so that the src pad can negotiate the proper caps on
time but this is far from ideal. Is there a way to properly solve this
problem?

Problem 2:
In case the input framerate is equal to the output framerate, things run
relatively smoothly using the workaround above. However, if the input
framerate is lower than the output framerate, I notice all kinds of visual
glitches in the output video. I am not an expert on the exact inner-workings
of the compositor element, but this is my assessment so far:

I change the output caps of videocrop src pad 10 times per second, but the
compositor element queries the sink pad linked to the videocrop element 25
times per second(?). Since I change the src pad caps just after a frame is
output on the src pad, the compositor element would (at the next interval)
query an incorrectly sized frame (because the src pad properties already
changed but no new frame was rendered on the videocrop src pad). This way,
the videocrop element expects a bigger/smaller frame on its sink pad than
actually available. Is this assessment correct or am I making a mistake?

Any insight on how to tackle this problem?

On a side note, perhaps problem 1 is related to the issue raised by Nicolas
Dufresne recently [2].

[1] https://www.dropbox.com/s/szavq5cbtrmtvj1/DynamicCropper.jpg?dl=0
[2]
http://gstreamer-devel.966125.n4.nabble.com/Re-gst-plugins-good-videocrop-mark-crop-properties-as-mutable-in-playing-state-td4677731.html




--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/videocrop-compositor-Change-output-of-videocrop-just-in-time-using-GstVideoCropMeta-metadata-tp4677790.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.