Image overlay over a video stream

[Wolfgang Grandegger]

Hello,

digging deeper about text, image and graphics overlay...

Am 18.10.2018 um 17:55 schrieb Nicolas Dufresne:
> Le jeudi 18 octobre 2018 à 16:21 +0200, Wolfgang Grandegger a écrit :
>> Hello,
>>
>> I'm currently evaluating the following pipeline to receive, display
>> and
>> record a MJPEG video stream:
>>
>>   # nice -20 \
>>     gst-launch-1.0 -v \
>>     udpsrc port=50004 buffer-size=180000000 do-timestamp=1 \
>>       caps="application/x-rtp, media=(string)video, clock-
>> rate=(int)90000, \
>>       encoding-name=(string)JPEG, payload=(int)26,
>> framerate=(fraction)50/1" \
>>     ! rtpjitterbuffer latency=20 \
>>     ! rtpjpegdepay \
>>     ! vaapijpegdec \
>>     ! timeoverlay \
>>     ! tee name=t
>>     t. ! queue ! vaapisink
>>     t. ! queue ! vaapih264enc ! mp4mux ! filesink
>> location=/tmp/test.mp4
>>
>> The CPU usage of the various threads with and without the element
>> "timeoverlay" is listed below:
>>
>>     with timeoverlay ->  no     yes 
>>   TID  Thread Name      CPU %  CPU %
>>   ----------------------------------
>>   542  gst-launch-1.0    42.0   88.8 
>>   550  queue0:src         0.2    0.4
>>   549  vaapiencodeh264    5.3    5.5
>>   548  gmain              0.0    0.0
>>   547  udpsrc0:src        8.9    8.4
>>   546  rtpjitterbuffer    8.5   51.6
>>   545  timer              0.0    0.0
>>   544  queue1:src        14.8   17.6
>>   543  queue0:src         3.9    4.9
>>
>> The "timeoverlay" adds approx. 45% to the CPU load (max is 4 x 100%).
>> What does take that much CPU time?
> 
> It's called software rendering (with anti-aliasing and all). There is
> also a hit because you need to download/upload the pixels from/to the
> GPU.

I see! The text needs to be rendered and inserted into (overlayed with)
the video frame. The CPU usage really depends how often and what is
rendered. Already disabling shadow or outline drawing or a smaller font
reduces the CPU load.

BTW, is it possible to specify the colour for the shaded background? I
know that it could be achieved with Pango "<span>" text attributes, e.g.
"bgcolor", but it requires more CPU time than the "shaded" background
from the text overlay.

>> Is there a faster way to do the overlay or is that element from the
>> Pango plugin already quiet efficient?
>> I want to overlay an image over the video stream, ideally done by
>> the graphics hardware.
> 
> There is an active effort to enable GL rendering of CompositonOverlay
> meta, but we don't have a fast method to import back GL textures into
> VAAPI encoder iirc. So there is still quite some work.

Is this work in progress visible somewhere, e.g. as GIT repo?

I realized, that the i.MX6 GStreamer-IMX Plugin [1] does have a somehow
optimized implementation of the "textoverlay" using 2D acceleration.
Would that approach be feasible and help on Intel graphics hardware as well?

Another option for text, image and graphics overlay is to use Cairo
directly using the "cairooverlay". That would allow to use the Cairo
text renderer and also add graphics or images in one process. Would that
be "lighter" or more efficient? I think GStreamer 0.1 did have a
"cairotextoverlay".

[1] https://github.com/Freescale/gstreamer-imx/src/g2d/pango/

Thanks for any input!

Wolfgang.