[Bug 110719] Crash in radeon_drv_video.so when attempting to convert rgb video data

[bugzilla-daemon at freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=110719

            Bug ID: 110719
           Summary: Crash in radeon_drv_video.so when attempting to
                    convert rgb video data
           Product: Mesa
           Version: git
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: medium
         Component: Drivers/Gallium/radeonsi
          Assignee: dri-devel at lists.freedesktop.org
          Reporter: oreaus at gmail.com
        QA Contact: dri-devel at lists.freedesktop.org

Kernel 5.0.0
Mesa 19.2.0-devel (git-629806b55b)
OpenGL renderer string: AMD Radeon (TM) RX 460 Graphics (POLARIS11, DRM 3.27.0,
5.0.0, LLVM 8.0.0)

I can reproduce the crash with this command:

`ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -i input.mp4 -vf
'format=rgb0,hwupload' -vcodec h264_vaapi -bf 0 output.mp4`

The input video does not have to be rgb0 format, this command just tells it
that it is to demonstrate the problem. av_hwframe_transfer_get_formats()
reports rgb0 is supported but clearly there's a problem. This is the backtrace
for h264_vaapi:

Thread 1 "ffmpeg" received signal SIGSEGV, Segmentation fault.
create (enc=0x555555ec73a0) at
../src/gallium/drivers/radeon/radeon_vce_52.c:188
188                     RVCE_CS(enc->chroma->u.legacy.level[0].nblk_x *
enc->chroma->bpe); // encRefPicChromaPitch
(gdb) bt full
#0  0x00007fffd740a407 in create (enc=0x555555ec73a0) at
../src/gallium/drivers/radeon/radeon_vce_52.c:188
        begin = 0x7fffc060102c
        sscreen = 0x555555812020
#1  0x00007fffd73bfe13 in rvce_begin_frame (encoder=0x555555ec73a0,
source=0x5555561c2ea0, picture=<optimized out>) at
../src/gallium/drivers/radeon/radeon_vce.c:291
        fb = {usage = 4, res = 0x555555ed02f0}
        enc = 0x555555ec73a0
        vid_buf = 0x5555561c2ea0
        pic = <optimized out>
        need_rate_control = true
#2  0x00007fffd72d32ed in vlVaEndPicture (ctx=<optimized out>,
context_id=<optimized out>) at ../src/gallium/state_trackers/va/picture.c:655
        drv = 0x55555580c6c0
        context = 0x555555ec3180
        coded_buf = 0x5555561e6360
        surf = 0x5555561f0320
        feedback = 0x555555a6eec0
        screen = <optimized out>
        supported = <optimized out>
        realloc = <optimized out>
        format = <optimized out>
#3  0x00007ffff085d820 in vaEndPicture () at
/usr/lib/x86_64-linux-gnu/libva.so.2
#4  0x00007ffff6212e53 in  () at /usr/lib/x86_64-linux-gnu/libavcodec.so.57
#5  0x00007ffff6213149 in  () at /usr/lib/x86_64-linux-gnu/libavcodec.so.57
#6  0x00007ffff62137be in  () at /usr/lib/x86_64-linux-gnu/libavcodec.so.57
#7  0x00007ffff5dbc6c7 in avcodec_encode_video2 () at
/usr/lib/x86_64-linux-gnu/libavcodec.so.57
#8  0x00007ffff5dbcaad in  () at /usr/lib/x86_64-linux-gnu/libavcodec.so.57
#9  0x00007ffff5dbcc5a in avcodec_send_frame () at
/usr/lib/x86_64-linux-gnu/libavcodec.so.57

and for hevc_vaapi:

Thread 1 "ffmpeg" received signal SIGSEGV, Segmentation fault.
0x00007fffd7406f4f in radeon_uvd_enc_encode_params_hevc (enc=0x555555ec7040) at
../src/gallium/drivers/radeon/radeon_uvd_enc_1_1.c:972
972        enc->enc_pic.enc_params.allowed_max_bitstream_size = enc->bs_size;
(gdb) bt full
#0  0x00007fffd7406f4f in radeon_uvd_enc_encode_params_hevc
(enc=0x555555ec7040) at ../src/gallium/drivers/radeon/radeon_uvd_enc_1_1.c:972
        sscreen = 0x555555812020
#1  0x00007fffd7406f4f in encode (enc=0x555555ec7040) at
../src/gallium/drivers/radeon/radeon_uvd_enc_1_1.c:1104
#2  0x00007fffd72d3304 in vlVaEndPicture (ctx=<optimized out>,
context_id=<optimized out>) at ../src/gallium/state_trackers/va/picture.c:656
        drv = 0x55555580c6c0
        context = 0x555555ec4420
        coded_buf = 0x555555ec6180
        surf = 0x5555561f0570
        feedback = 0x5555557bd0c0
        screen = <optimized out>
        supported = <optimized out>
        realloc = <optimized out>
        format = <optimized out>
#3  0x00007ffff085d820 in vaEndPicture () at
/usr/lib/x86_64-linux-gnu/libva.so.2
#4  0x00007ffff6212e53 in  () at /usr/lib/x86_64-linux-gnu/libavcodec.so.57
#5  0x00007ffff6213149 in  () at /usr/lib/x86_64-linux-gnu/libavcodec.so.57
#6  0x00007ffff62137be in  () at /usr/lib/x86_64-linux-gnu/libavcodec.so.57
#7  0x00007ffff5dbc6c7 in avcodec_encode_video2 () at
/usr/lib/x86_64-linux-gnu/libavcodec.so.57
#8  0x00007ffff5dbcaad in  () at /usr/lib/x86_64-linux-gnu/libavcodec.so.57
#9  0x00007ffff5dbcc5a in avcodec_send_frame () at
/usr/lib/x86_64-linux-gnu/libavcodec.so.57

It seems the driver should support this format or at least just report only the
formats it actually supports.

Please note that this bug was discovered by a screen recorder implementing
vaapi. Passing rgb data to the intel vaapi driver works as expected.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/dri-devel/attachments/20190521/21c6e3e4/attachment.html>