[Mesa-dev] [mesa-20.3.2] NULL pointer dereference in vl_compositor_yuv_deint_full
Alexander Kapshuk
alexander.kapshuk at gmail.com
Tue Jan 5 07:51:18 UTC 2021
On Mon, Jan 4, 2021 at 11:02 PM Thong Thai <thong.thai at amd.com> wrote:
>
> Hi Alexander,
>
> Hope this fixes your problem - VLC should give up trying VAAPI and it'll
> try using VDPAU or revert to software rendering.
>
> https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8320
>
> Regards,
>
> Thong Thai
>
> On 2021-01-04 2:21 p.m., Alexander Kapshuk wrote:
> > On Mon, Jan 4, 2021 at 9:03 PM Thong Thai <thong.thai at amd.com> wrote:
> >> Hi Alexander,
> >>
> >> Guess I should add a check to see if it can create a non-interlaced
> >> video buffer, and if not, return what it would have returned before,
> >> when trying to derive an image from an interlaced buffer:
> >>
> >> return VA_STATUS_ERROR_OPERATION_FAILED;
> >>
> >> Thanks,
> >>
> >> Thong Thai
> >>
> >> On 2021-01-03 3:38 p.m., Alexander Kapshuk wrote:
> >>> NVIDIA chip affected:
> >>> 01:00.0 VGA compatible controller: NVIDIA Corporation GT216 [GeForce
> >>> 210] (rev a1)
> >>>
> >>> The null pointer dereference occurs here:
> >>> Thread 27 "vlc" received signal SIGSEGV, Segmentation fault.
> >>> [Switching to Thread 0x7fff8f7c1640 (LWP 79292)]
> >>> 0x00007fff8d59d1da in vl_compositor_yuv_deint_full (s=0x7fff980e8518,
> >>> c=0x7fff980e83d8, src=0x7fff98670030, dst=0x0,
> >>> src_rect=0x7fff8f7c0470, dst_rect=0x7fff8f7c0460,
> >>> deinterlace=VL_COMPOSITOR_WEAVE) at
> >>> ../mesa-20.3.2/src/gallium/auxiliary/vl/vl_compositor.c:689
> >>> 689 dst_surfaces = dst->get_surfaces(dst); //dst==NULL
> >>>
> >>> => 0x00007fff8d5981da <+42>: call *0x38(%rcx) //rcx is dst
> >>> (gdb) i r rcx
> >>> rcx 0x0 0
> >>>
> >>> (gdb) bt
> >>> #0 0x00007fff8d59d1da in vl_compositor_yuv_deint_full
> >>> (s=0x7fff980e8518, c=0x7fff980e83d8, src=0x7fff98670030, dst=0x0,
> >>> src_rect=0x7fff8f7c0470, dst_rect=0x7fff8f7c0460,
> >>> deinterlace=VL_COMPOSITOR_WEAVE) at
> >>> ../mesa-20.3.2/src/gallium/auxiliary/vl/vl_compositor.c:689
> >>> #1 0x00007fff8d58a29b in vlVaDeriveImage (ctx=0x7fff980c1590,
> >>> surface=<optimized out>, image=0x7fff8f7c05e0) at
> >>> ../mesa-20.3.2/src/gallium/frontends/va/image.c:321
> >>> #2 0x00007fff91485799 in vaDeriveImage () at /usr/lib/libva.so.2
> >>> #3 0x00007fff8e2256d2 in () at
> >>> /usr/lib/vlc/plugins/video_output/libglconv_vaapi_x11_plugin.so
> >>> #4 0x00007fff8e224189 in () at
> >>> /usr/lib/vlc/plugins/video_output/libglconv_vaapi_x11_plugin.so
> >>> #5 0x00007fff8f6b1896 in () at
> >>> /usr/lib/vlc/plugins/video_output/libgl_plugin.so
> >>> #6 0x00007fff8f6b86db in () at
> >>> /usr/lib/vlc/plugins/video_output/libgl_plugin.so
> >>> #7 0x00007ffff7d07cee in () at /usr/lib/libvlccore.so.9
> >>> #8 0x00007ffff7cfa019 in () at /usr/lib/libvlccore.so.9
> >>> #9 0x00007ffff7cfbf9e in () at /usr/lib/libvlccore.so.9
> >>> #10 0x00007ffff7f623e9 in start_thread () at /usr/lib/libpthread.so.0
> >>> #11 0x00007ffff7e8a293 in clone () at /usr/lib/libc.so.6
> >>>
> >>> mesa-20.3.2/src/gallium/frontends/va/image.c:312,313
> >>> VAStatus
> >>> vlVaDeriveImage(VADriverContextP ctx, VASurfaceID surface, VAImage *image)
> >>> {
> >>> ...
> >>> new_template.interlaced = false; //create_video_buffer
> >>> returns NULL if new_template.interlaced is set to false See below.
> >>> new_buffer = drv->pipe->create_video_buffer(drv->pipe, &new_template);
> >>> ...
> >>> vl_compositor_yuv_deint_full(&drv->cstate, &drv->compositor,
> >>> surf->buffer, new_buffer,
> >>> &src_rect, &dst_rect,
> >>> VL_COMPOSITOR_WEAVE);
> >>> ...
> >>> }
> >>>
> >>> mesa-20.3.2/src/gallium/drivers/nouveau/nv50/nv84_video.c:618,621
> >>> struct pipe_video_buffer *
> >>> nv84_video_buffer_create(struct pipe_context *pipe,
> >>> const struct pipe_video_buffer *template)
> >>> {
> >>> ...
> >>> if (!template->interlaced) { //set to false in vlVaDeriveImage. See above
> >>> debug_printf("Require interlaced video buffers\n");
> >>> return NULL;
> >>> }
> >>> ...
> >>> }
> >>>
> >>> Here's the commit that introduced the null pointer dereference in
> >>> question, https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fmesa%2Fmesa%2F-%2Fcommit%2Ffcb558321e65b62244a11e0066bb8713b1854721&data=04%7C01%7Cthong.thai%40amd.com%7C7bd14f7e1d4c4bce7aa908d8b0e5fdff%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637453849143396990%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aCtzRZ4jy7OHXYrYeQJyLN%2BIDnx%2F7g530vTAvY8gAd0%3D&reserved=0.
> >>> Please advise on the further course of action.
> >>> _______________________________________________
> >>> mesa-dev mailing list
> >>> mesa-dev at lists.freedesktop.org
> >>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fmesa-dev&data=04%7C01%7Cthong.thai%40amd.com%7C7bd14f7e1d4c4bce7aa908d8b0e5fdff%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637453849143406947%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=b1IKXd9bCpa8yTXQ%2FBL%2FI0KFWwYQTVTPywsTl7CS3%2FM%3D&reserved=0
> >> _______________________________________________
> >> mesa-dev mailing list
> >> mesa-dev at lists.freedesktop.org
> >> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fmesa-dev&data=04%7C01%7Cthong.thai%40amd.com%7C7bd14f7e1d4c4bce7aa908d8b0e5fdff%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637453849143406947%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=b1IKXd9bCpa8yTXQ%2FBL%2FI0KFWwYQTVTPywsTl7CS3%2FM%3D&reserved=0
> > Thanks.
Hi Thong,
Thanks for taking prompt action on this.
This is to confirm that your proposed patch does fix the null pointer
dereference error for me. I have tested this in my local build.
Please find below a minor suggestion to eliminate repetitive code.
--- mesa-20.3.2/src/gallium/frontends/va/image_orig.c
2021-01-05 09:21:19.087955994 +0200
+++ mesa-20.3.2/src/gallium/frontends/va/image.c 2021-01-05
09:19:45.032308235 +0200
@@ -202,0 +203 @@
+ VAStatus status;
@@ -317,3 +318,2 @@
- FREE(img);
- mtx_unlock(&drv->mutex);
- return VA_STATUS_ERROR_OPERATION_FAILED;
+ status = VA_STATUS_ERROR_OPERATION_FAILED;
+ goto exit_on_error;
@@ -356,3 +356,2 @@
- FREE(img);
- mtx_unlock(&drv->mutex);
- return VA_STATUS_ERROR_OPERATION_FAILED;
+ status = VA_STATUS_ERROR_OPERATION_FAILED;
+ goto exit_on_error;
@@ -363,3 +362,2 @@
- FREE(img);
- mtx_unlock(&drv->mutex);
- return VA_STATUS_ERROR_ALLOCATION_FAILED;
+ status = VA_STATUS_ERROR_ALLOCATION_FAILED;
+ goto exit_on_error;
@@ -382,0 +381,5 @@
+
+exit_on_error:
+ FREE(img);
+ mtx_unlock(&drv->mutex);
+ return status;
More information about the mesa-dev
mailing list