[Mesa-dev] [Bug 110673] amdgpu hevc encoding problems: segment fault and contents of garbage

[bugzilla-daemon at freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=110673

            Bug ID: 110673
           Summary: amdgpu hevc encoding problems: segment fault and
                    contents of garbage
           Product: Mesa
           Version: unspecified
          Hardware: All
                OS: Linux (All)
            Status: NEW
          Severity: normal
          Priority: medium
         Component: Other
          Assignee: mesa-dev at lists.freedesktop.org
          Reporter: mfk530 at gmail.com
        QA Contact: mesa-dev at lists.freedesktop.org

I have tested amdgpu hevc encoding via libva few days ago.
And I found two problems.

Hardware: R5 2400g(VCN), rx580/vega64(UVD_ENC)

1. intra_period must >= 8 otherwise segment fault in mesa.

I found these codes in mesa using gdb:

radeon_uvd_enc.c/radeon_vcn_enc.c:

enc->enc_pic.max_poc = pic->seq.intra_period;
for (int i = enc->enc_pic.max_poc; i != 0; enc->enc_pic.log2_max_poc++)
    i = (i >> 1);

radeon_uvd_enc_1_1.c/radeon_vcn_enc_1_2.c:

radeon_uvd_enc_code_ue(enc, enc->enc_pic.log2_max_poc - 4);
radeon_enc_code_ue(enc, enc->enc_pic.log2_max_poc - 4);

If intra_period < 8, (enc->enc_pic.log2_max_poc - 4) is negative.
cs array will overflow in radeon_uvd_enc_code_ue or radeon_enc_code_ue.

ffmpeg can reproduce this problem by changing "-g" parameter value.

ffmpeg -vaapi_device /dev/dri/renderD128 -i input.avi -vf
'format=nv12,hwupload' -c:v hevc_vaapi -bf 0 -g 7 output.mp4

2. Hevc encoding only works correctly in CQP mode.

I have tested CBR and VBR, both of them produce contents of garbage， green or
black mostly.

ffmpeg can reproduce this problem.

ffmpeg -vaapi_device /dev/dri/renderD128 -i input.avi -vf
'format=nv12,hwupload' -c:v hevc_vaapi -bf 0 -b:v 5M output.mp4

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/mesa-dev/attachments/20190514/fd30b5bf/attachment-0001.html>