Image overlay over a video stream
wg at grandegger.com
Thu Nov 15 08:46:58 UTC 2018
digging deeper about text, image and graphics overlay...
Am 18.10.2018 um 17:55 schrieb Nicolas Dufresne:
> Le jeudi 18 octobre 2018 à 16:21 +0200, Wolfgang Grandegger a écrit :
>> I'm currently evaluating the following pipeline to receive, display
>> record a MJPEG video stream:
>> # nice -20 \
>> gst-launch-1.0 -v \
>> udpsrc port=50004 buffer-size=180000000 do-timestamp=1 \
>> caps="application/x-rtp, media=(string)video, clock-
>> rate=(int)90000, \
>> encoding-name=(string)JPEG, payload=(int)26,
>> framerate=(fraction)50/1" \
>> ! rtpjitterbuffer latency=20 \
>> ! rtpjpegdepay \
>> ! vaapijpegdec \
>> ! timeoverlay \
>> ! tee name=t
>> t. ! queue ! vaapisink
>> t. ! queue ! vaapih264enc ! mp4mux ! filesink
>> The CPU usage of the various threads with and without the element
>> "timeoverlay" is listed below:
>> with timeoverlay -> no yes
>> TID Thread Name CPU % CPU %
>> 542 gst-launch-1.0 42.0 88.8
>> 550 queue0:src 0.2 0.4
>> 549 vaapiencodeh264 5.3 5.5
>> 548 gmain 0.0 0.0
>> 547 udpsrc0:src 8.9 8.4
>> 546 rtpjitterbuffer 8.5 51.6
>> 545 timer 0.0 0.0
>> 544 queue1:src 14.8 17.6
>> 543 queue0:src 3.9 4.9
>> The "timeoverlay" adds approx. 45% to the CPU load (max is 4 x 100%).
>> What does take that much CPU time?
> It's called software rendering (with anti-aliasing and all). There is
> also a hit because you need to download/upload the pixels from/to the
I see! The text needs to be rendered and inserted into (overlayed with)
the video frame. The CPU usage really depends how often and what is
rendered. Already disabling shadow or outline drawing or a smaller font
reduces the CPU load.
BTW, is it possible to specify the colour for the shaded background? I
know that it could be achieved with Pango "<span>" text attributes, e.g.
"bgcolor", but it requires more CPU time than the "shaded" background
from the text overlay.
>> Is there a faster way to do the overlay or is that element from the
>> Pango plugin already quiet efficient?
>> I want to overlay an image over the video stream, ideally done by
>> the graphics hardware.
> There is an active effort to enable GL rendering of CompositonOverlay
> meta, but we don't have a fast method to import back GL textures into
> VAAPI encoder iirc. So there is still quite some work.
Is this work in progress visible somewhere, e.g. as GIT repo?
I realized, that the i.MX6 GStreamer-IMX Plugin  does have a somehow
optimized implementation of the "textoverlay" using 2D acceleration.
Would that approach be feasible and help on Intel graphics hardware as well?
Another option for text, image and graphics overlay is to use Cairo
directly using the "cairooverlay". That would allow to use the Cairo
text renderer and also add graphics or images in one process. Would that
be "lighter" or more efficient? I think GStreamer 0.1 did have a
Thanks for any input!
More information about the gstreamer-devel