[Libva] Weird memory problem around vaGetDisplay

[Ratin]

# uname -a
Linux nuc1 3.16.0-51-generic #69~14.04.1-Ubuntu SMP Wed Oct 7 15:32:41 UTC
2015 x86_64 x86_64 x86_64 GNU/Linux




On Wed, Oct 21, 2015 at 12:13 PM, Ratin <ratin3 at gmail.com> wrote:

> Breakpoint 1, vaGetDisplay (native_dpy=0x7fffe4312870) at va_x11.c:154
> 154    {
> (gdb) list
> 149
> 150
> 151    VADisplay vaGetDisplay (
> 152        Display *native_dpy /* implementation specific */
> 153    )
> 154    {
> 155      VADisplay dpy = NULL;
> 156      VADisplayContextP pDisplayContext;
> 157
> ;;;
> 179          pDisplayContext->opaque          = NULL;
> 177          pDisplayContext->vaDestroy       = va_DisplayContextDestroy;
> 178          pDisplayContext->vaGetDriverName =
> va_DisplayContextGetDriverName;
> 181          dpy                              = (VADisplay)pDisplayContext;
> 195    }
> (gdb) p dpy
> $2 = (VADisplay) 0x7fffe42d3160
> (gdb) n <========================== stack unroll happens here
> (gdb) list
> 935            pVAAPICNTX->vaDisplay = (VADisplay)  <==== upper level code
> vaGetDisplay(pVAAPICNTX->mX11Display);
> 936             pVAAPICNTX->voWindow = pRendererParams->mWindow;
> (gdb) p pVAAPICNTX->vaDisplay
> $3 = (void *) 0xffffffffe42d3160
>
>
> pVAAPICNTX is malloced struct ptr and one of the element of it is the
> vaDisplay declated like this :
>
> typedef struct _VAAPICNTX
> {
>     uint32_t                  VAContextID;
>    Window                   voWindow;
>    int voWidth;
>    int voHeight;
> ;;;
> ;;;
>     VADisplay               vaDisplay;
>
> ;;;
>
> ;;;
>
> }VAAPICNTX, *PVAAPICNTX;
>
>
> As you can see, dpy above around line 195 in  vaGetDisplay function is
> shown as 0x7fffe42d3160 but once the program counter comes out of that
> function and stack rolling happens, and context switches to upper level
> code, the assigned value has  0xffffffffe42d3160 (sign extended vs
> unsigned). What might cause this ? Next instance when I call vaInitialize
> with this vaDisplay, it craches:
>
> Program received signal SIGSEGV, Segmentation fault.
> vaDisplayIsValid (dpy=0xffffffffe42d3450) at va.c:102
> 102        return pDisplayContext && (pDisplayContext->vadpy_magic ==
> VA_DISPLAY_MAGIC) && pDisplayContext->vaIsValid(pDisplayContext);
>
>
>
> Any help/pointer would be appreciated. Thanks
>
>
>
> gcc --version
> gcc (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4
> Copyright (C) 2013 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
> ls -la libc-2.19.so
> -rwxr-xr-x 1 root root 1845024 Dec  4  2014 libc-2.19.so
>
> ldd --version
> ldd (Ubuntu EGLIBC 2.19-0ubuntu6.5) 2.19
>
>
> Platform:
>
> Ratin
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/libva/attachments/20151021/9c73ba2f/attachment.html>