[Mesa-users] [vaapi AMD VCE] Hardware encoder outputs messed up video

[Rob McCathie]

On 04/07/17 20:37, Rob McCathie wrote:
> Hello Mesa Users list,
>
> I have an AMD A10-7800 APU ( Steamroller/Kaveri/VCE2.0) and would like 
> to use the h264 (MPEG4 AVC) hardware encoding capability that 
> Xorg/Mesa documentation states is supported 
> (https://www.x.org/wiki/RadeonFeature/#index9h2).
>
> I've been making attempts with both gstreamer (gst-launch-1.0) and 
> ffmpeg but am having the same problematic outcome with both.
> I am performing the colourspace conversion to NV12 that research leads 
> me to believe is required (and indeed gst-launch does not succeed if 
> this conversion is not done). (...though Wikipedia says VCE 2.0 also 
> supports YUV444 and i'd like to be able to use it, but one thing at a 
> time.)
>
> Here is a video of the gstreamer test pattern encoded using a software 
> encoder (x264), it's produced with this following command:
> gst-launch-1.0 videotestsrc num-buffers=200 ! video/x-raw,format=NV12 
> ! x264enc ! video/x-h264,profile=main ! h264parse ! matroskamux ! 
> filesink location=test-x264enc.mkv
> http://paradoxcomputers.com.au/stuff/amd_vaapi_issue/test-x264enc.mkv
> It looks how it should.
>
> Here's the outcome when i try to use hardware h264 encoding with 
> VA-API, produced with this following command:
> gst-launch-1.0 videotestsrc num-buffers=200 ! video/x-raw,format=NV12 
> ! vaapih264enc ! video/x-h264,profile=main ! h264parse ! matroskamux ! 
> filesink location=test-vaapih264enc.mkv
> http://paradoxcomputers.com.au/stuff/amd_vaapi_issue/test-vaapih264enc.mkv 
>
> It's all messed up, the image is duplicated (above and below) and the 
> colours are wonky (like the colour channels are separated and all 
> offset from each other... or something).
>
> I've tested this on Debian 9 (Mesa 13.0.6) and an up-to-date Arch 
> Linux install (Mesa 17.1.4) and get the same result on both. On the 
> Arch Linux install I've also tried using AMDGPU driver rather than 
> radeon - same result. (well actually, the output on AMDGPU did seem a 
> little different, but messed up in a similar fashion nonetheless).
>
> Any help would be appreciated.
>
>

I just asked in #dri-devel on Freenode IRC and they were able to help 
me. Needed to set environment variable: VAAPI_DISABLE_INTERLACE=1
Thanks jkqxz on IRC!


--
Regards,

Rob McCathie