gstreamer-vaapi: vaapipostproc and videomixer

[Sebastian Dröge]

On Mo, 2014-04-28 at 14:43 -0700, Joe McDonald wrote:
> Hey all,
> 
> I've been looking pretty intently for a few days now into how I can get GPU
> hardware acceleration on my Atom E3845 dev board when implementing a
> multiple source tiled video output and I'm running into something of a stone
> wall here.  
> 
> For reference see the following bugs in the gstreamer-vaapi:
> Bug 720174 <https://bugzilla.gnome.org/show_bug.cgi?id=720174>  
> Bug 720311 <https://bugzilla.gnome.org/show_bug.cgi?id=720311>  
> 
> What I want to know is if anyone has gotten around the limitation of
> vaapipostproc such that something like videomixer could be tied into the
> pipeline in a functional manner.  
> 
> A quick example of what I'd like to do (without hw decoding):
> gst-launch-1.0 -v \
>     videomixer \
>         sink_0::xpos=0 sink_0::ypos=0 \ 
>         sink_1::xpos=640 sink_1::ypos=0 \ 
>         sink_2::xpos=0 sink_2::ypos=360 \    
>         sink_3::xpos=640 sink_3::ypos=360 \ name=mixer ! vaapisink \
>     filesrc location=$1 ! mpegpsdemux ! mpegvideoparse ! mpeg2dec !
> videoscale ! video/x-raw,width=640,height=360 ! mixer. \ 
>     filesrc location=$1 ! mpegpsdemux ! mpegvideoparse ! mpeg2dec !
> videoscale ! video/x-raw,width=640,height=360 ! mixer. \ 
>     filesrc location=$1 ! mpegpsdemux ! mpegvideoparse ! mpeg2dec !
> videoscale ! video/x-raw,width=640,height=360 ! mixer. \ 
>     filesrc location=$1 ! mpegpsdemux ! mpegvideoparse ! mpeg2dec !
> videoscale ! video/x-raw,width=640,height=360 ! mixer.
> 
> While this will run, the CPU utilization on this little processor goes
> skyhigh.  If I replace the 'mpeg2dec' and 'videoscale' with the appropriate
> 'vaapidecode' and 'vaapipostproc width=640' I get nothing but nasty green
> screen output on the display.  From what I've read this is due to the
> misalignment of the output format from vaapipostproc
> (video/x-raw(memory:VASurface)) and what videomixer expects(pure
> video/x-raw). 
> 
> Some additional info.  In the 0.5.8 version of gstreamer-vaapi there is some
> source code referencing the vaapidownload functions that used to exist in
> previous releases (compatible with gstreamer-0.10) that would allow for the
> conversion from a VASurface to something usable generally by gstreamer.  I'm
> not sure if there's any way that I can implement this in an application, but
> if anyone can point me in the right direction it'd be greatly appreciated.  

The problem here will be that videomixer only works on raw video in
normal memory and not on vaapi surfaces. There are some bugs around that
in gst-vaapi as you've already noticed, especially for making the usage
of vaapi surfaces completely transparent in GStreamer pipelines.

But apart from that you will probably still get very high CPU usage if
all this works as videomixer works completely in software. A better
solution would be to do the compositing/mixing either with vaapi
somehow, or by rendering everything into GL textures and then composite
everything together in GL (with transferring the vaapi surfaces
direectly into GL textures and never ever copy them to normal system
memory).

-- 
Sebastian Dröge, Centricular Ltd - http://www.centricular.com
Expertise, Straight from the Source
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 966 bytes
Desc: This is a digitally signed message part
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20140506/7aaf7be4/attachment-0001.sig>