[Mesa-dev] [mesa-20.3.2] NULL pointer dereference in vl_compositor_yuv_deint_full

[Alexander Kapshuk]

On Mon, Jan 4, 2021 at 9:03 PM Thong Thai <thong.thai at amd.com> wrote:
>
> Hi Alexander,
>
> Guess I should add a check to see if it can create a non-interlaced
> video buffer, and if not, return what it would have returned before,
> when trying to derive an image from an interlaced buffer:
>
> return VA_STATUS_ERROR_OPERATION_FAILED;
>
> Thanks,
>
> Thong Thai
>
> On 2021-01-03 3:38 p.m., Alexander Kapshuk wrote:
> > NVIDIA chip affected:
> > 01:00.0 VGA compatible controller: NVIDIA Corporation GT216 [GeForce
> > 210] (rev a1)
> >
> > The null pointer dereference occurs here:
> > Thread 27 "vlc" received signal SIGSEGV, Segmentation fault.
> > [Switching to Thread 0x7fff8f7c1640 (LWP 79292)]
> > 0x00007fff8d59d1da in vl_compositor_yuv_deint_full (s=0x7fff980e8518,
> > c=0x7fff980e83d8, src=0x7fff98670030, dst=0x0,
> > src_rect=0x7fff8f7c0470, dst_rect=0x7fff8f7c0460,
> > deinterlace=VL_COMPOSITOR_WEAVE) at
> > ../mesa-20.3.2/src/gallium/auxiliary/vl/vl_compositor.c:689
> > 689     dst_surfaces = dst->get_surfaces(dst); //dst==NULL
> >
> > => 0x00007fff8d5981da <+42>:    call   *0x38(%rcx) //rcx is dst
> > (gdb) i r rcx
> > rcx            0x0                 0
> >
> > (gdb) bt
> > #0  0x00007fff8d59d1da in vl_compositor_yuv_deint_full
> > (s=0x7fff980e8518, c=0x7fff980e83d8, src=0x7fff98670030, dst=0x0,
> > src_rect=0x7fff8f7c0470, dst_rect=0x7fff8f7c0460,
> > deinterlace=VL_COMPOSITOR_WEAVE) at
> > ../mesa-20.3.2/src/gallium/auxiliary/vl/vl_compositor.c:689
> > #1  0x00007fff8d58a29b in vlVaDeriveImage (ctx=0x7fff980c1590,
> > surface=<optimized out>, image=0x7fff8f7c05e0)    at
> > ../mesa-20.3.2/src/gallium/frontends/va/image.c:321
> > #2  0x00007fff91485799 in vaDeriveImage () at /usr/lib/libva.so.2
> > #3  0x00007fff8e2256d2 in  () at
> > /usr/lib/vlc/plugins/video_output/libglconv_vaapi_x11_plugin.so
> > #4  0x00007fff8e224189 in  () at
> > /usr/lib/vlc/plugins/video_output/libglconv_vaapi_x11_plugin.so
> > #5  0x00007fff8f6b1896 in  () at
> > /usr/lib/vlc/plugins/video_output/libgl_plugin.so
> > #6  0x00007fff8f6b86db in  () at
> > /usr/lib/vlc/plugins/video_output/libgl_plugin.so
> > #7  0x00007ffff7d07cee in  () at /usr/lib/libvlccore.so.9
> > #8  0x00007ffff7cfa019 in  () at /usr/lib/libvlccore.so.9
> > #9  0x00007ffff7cfbf9e in  () at /usr/lib/libvlccore.so.9
> > #10 0x00007ffff7f623e9 in start_thread () at /usr/lib/libpthread.so.0
> > #11 0x00007ffff7e8a293 in clone () at /usr/lib/libc.so.6
> >
> > mesa-20.3.2/src/gallium/frontends/va/image.c:312,313
> > VAStatus
> > vlVaDeriveImage(VADriverContextP ctx, VASurfaceID surface, VAImage *image)
> > {
> > ...
> >           new_template.interlaced = false; //create_video_buffer
> > returns NULL if new_template.interlaced is set to false See below.
> >           new_buffer = drv->pipe->create_video_buffer(drv->pipe, &new_template);
> > ...
> >           vl_compositor_yuv_deint_full(&drv->cstate, &drv->compositor,
> >                             surf->buffer, new_buffer,
> >                             &src_rect, &dst_rect,
> >                             VL_COMPOSITOR_WEAVE);
> > ...
> > }
> >
> > mesa-20.3.2/src/gallium/drivers/nouveau/nv50/nv84_video.c:618,621
> > struct pipe_video_buffer *
> > nv84_video_buffer_create(struct pipe_context *pipe,
> >                           const struct pipe_video_buffer *template)
> > {
> > ...
> >     if (!template->interlaced) { //set to false in vlVaDeriveImage. See above
> >        debug_printf("Require interlaced video buffers\n");
> >        return NULL;
> >     }
> > ...
> > }
> >
> > Here's the commit that introduced the null pointer dereference in
> > question, https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fmesa%2Fmesa%2F-%2Fcommit%2Ffcb558321e65b62244a11e0066bb8713b1854721&data=04%7C01%7Cthong.thai%40amd.com%7C05879305275d42ce31ff08d8b0279037%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637453031560482800%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Cx3j7nIJU5Qoc%2BNpP5UTdRHJyS7hguLbWmoJsybTu0Q%3D&reserved=0.
> > Please advise on the further course of action.
> > _______________________________________________
> > mesa-dev mailing list
> > mesa-dev at lists.freedesktop.org
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fmesa-dev&data=04%7C01%7Cthong.thai%40amd.com%7C05879305275d42ce31ff08d8b0279037%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637453031560482800%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=QqIG2scFQxNzQFG04%2F5NRihMO6pgivupkCoHzGoaUOM%3D&reserved=0
> _______________________________________________
> mesa-dev mailing list
> mesa-dev at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/mesa-dev

Thanks.