Trying to understand how to make alpha & ximagesrc work efficiently

[Alain Culos]

Hi,

I am a recent newcomer to gstreamer.
Trying to use `gst-launch-1.0` to chroma-key my webcam and insert a pdf as background.

My attempt is very imperfect:
1/ I have not been able to find the right parameters to make the window sizes match.
2/ gst-launch occasionally complains it loses a lot of buffers.
3/ the video output only updates seemingly randomly, infrequently, and seems to (but not always) needs the pdf window to have focus and the mouse to be over it (but not exclusively).

Maybe there is a better solution.
My desktop is quite powerful: a modern (about 1yo, i7, 6-core, hyperthreaded, 64GB RAM), 4k display, 1080p USB webcam, SSD.
Linux 5.3.0-40-generic #32~18.04.1-Ubuntu SMP Mon Feb 3 14:05:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
()

The commands I ran are:

1/ testing the chroma-key with a chequered background
gst-launch-1.0 \
  videotestsrc pattern=checkers-8 ! video/x-raw,width=800,height=640 ! mixer.sink_0 \
  autovideosrc ! video/x-raw,width=800 ! alpha method=custom target-r=255 target-g=255 target-b=255 ! mixer.sink_1 \
  videomixer name=mixer sink_0::zorder=0 sink_1::zorder=1 ! videoconvert ! autovideosink
-> Very slow video, does not chroma-key on white!
gst-launch recurrent message: "A lot of buffers are being dropped"

2/ testing the chroma-key (2nd attempt)
gst-launch-1.0 \
  videotestsrc pattern=checkers-8 ! video/x-raw,width=800,height=640,framerate=1/1 ! mixer.sink_0 \
  autovideosrc ! video/x-raw,width=800 ! alpha method=custom target-r=255 target-g=255 target-b=255 ! mixer.sink_1 \
  videomixer name=mixer sink_0::zorder=0 sink_1::zorder=1 ! videoconvert ! autovideosink
-> Same as above

3/ testing my video
gst-launch-1.0 autovideosrc ! video/x-raw,width=800 ! videoconvert ! autovideosink
-> Showing my webcam is almost real-time, but has a bit of lag
The same test at 1080p is very laggy

4/ testing a source application window (evince, showing a pdf file)
gst-launch-1.0 \
  ximagesrc xid=0x06800008 ! mixer.sink_0 \
  autovideosrc ! video/x-raw,width=800 ! alpha method=custom target-r=255 target-g=255 target-b=255 ! mixer.sink_1 \
  videomixer name=mixer sink_0::zorder=0 sink_1::zorder=1 ! videoconvert ! autovideosink
-> Does not update the video in realtime (only updates rarely, at most every few seconds, when the application window has the focus or redraw events (possibly and/or mouse over)), does not allow me to resize the application window video stream.


Hoping to find here some pointers to solve the issues highlighted above, most importantly performance (framerate).

Next steps: a/ including the audio stream, b/ outputting to a video device so I can use this as a source for a video conference (zoom/skype/jitsi), c/ using an actual green screen.

Thanks for any help, pointers to documentation or tutorials.
Regards,
Alain.