[Mesa-users] [vaapi AMD VCE] Hardware encoder outputs messed up video

[Rob McCathie]

Hello Mesa Users list,

I have an AMD A10-7800 APU ( Steamroller/Kaveri/VCE2.0) and would like 
to use the h264 (MPEG4 AVC) hardware encoding capability that Xorg/Mesa 
documentation states is supported 
(https://www.x.org/wiki/RadeonFeature/#index9h2).

I've been making attempts with both gstreamer (gst-launch-1.0) and 
ffmpeg but am having the same problematic outcome with both.
I am performing the colourspace conversion to NV12 that research leads 
me to believe is required (and indeed gst-launch does not succeed if 
this conversion is not done). (...though Wikipedia says VCE 2.0 also 
supports YUV444 and i'd like to be able to use it, but one thing at a time.)

Here is a video of the gstreamer test pattern encoded using a software 
encoder (x264), it's produced with this following command:
gst-launch-1.0 videotestsrc num-buffers=200 ! video/x-raw,format=NV12 ! 
x264enc ! video/x-h264,profile=main ! h264parse ! matroskamux ! filesink 
location=test-x264enc.mkv
http://paradoxcomputers.com.au/stuff/amd_vaapi_issue/test-x264enc.mkv
It looks how it should.

Here's the outcome when i try to use hardware h264 encoding with VA-API, 
produced with this following command:
gst-launch-1.0 videotestsrc num-buffers=200 ! video/x-raw,format=NV12 ! 
vaapih264enc ! video/x-h264,profile=main ! h264parse ! matroskamux ! 
filesink location=test-vaapih264enc.mkv
http://paradoxcomputers.com.au/stuff/amd_vaapi_issue/test-vaapih264enc.mkv
It's all messed up, the image is duplicated (above and below) and the 
colours are wonky (like the colour channels are separated and all offset 
from each other... or something).

I've tested this on Debian 9 (Mesa 13.0.6) and an up-to-date Arch Linux 
install (Mesa 17.1.4) and get the same result on both. On the Arch Linux 
install I've also tried using AMDGPU driver rather than radeon - same 
result. (well actually, the output on AMDGPU did seem a little 
different, but messed up in a similar fashion nonetheless).

Any help would be appreciated.


-- 
Regards,

Rob McCathie