[Libva] Weird memory problem around vaGetDisplay

[Ratin]

Found the culprit, it was missing header (libva_x11.h). If anybody
interested to know, this caused the compiler to think the function will
return a 32 bit pointer, so the compiler inserted a cltq statement into the
assembly code which was flipping the upper 32 bits to all 1s.

On Wed, Oct 21, 2015 at 12:15 PM, Ratin <ratin3 at gmail.com> wrote:

> # uname -a
> Linux nuc1 3.16.0-51-generic #69~14.04.1-Ubuntu SMP Wed Oct 7 15:32:41 UTC
> 2015 x86_64 x86_64 x86_64 GNU/Linux
>
>
>
>
> On Wed, Oct 21, 2015 at 12:13 PM, Ratin <ratin3 at gmail.com> wrote:
>
>> Breakpoint 1, vaGetDisplay (native_dpy=0x7fffe4312870) at va_x11.c:154
>> 154    {
>> (gdb) list
>> 149
>> 150
>> 151    VADisplay vaGetDisplay (
>> 152        Display *native_dpy /* implementation specific */
>> 153    )
>> 154    {
>> 155      VADisplay dpy = NULL;
>> 156      VADisplayContextP pDisplayContext;
>> 157
>> ;;;
>> 179          pDisplayContext->opaque          = NULL;
>> 177          pDisplayContext->vaDestroy       = va_DisplayContextDestroy;
>> 178          pDisplayContext->vaGetDriverName =
>> va_DisplayContextGetDriverName;
>> 181          dpy                              =
>> (VADisplay)pDisplayContext;
>> 195    }
>> (gdb) p dpy
>> $2 = (VADisplay) 0x7fffe42d3160
>> (gdb) n <========================== stack unroll happens here
>> (gdb) list
>> 935            pVAAPICNTX->vaDisplay = (VADisplay)  <==== upper level
>> code vaGetDisplay(pVAAPICNTX->mX11Display);
>> 936             pVAAPICNTX->voWindow = pRendererParams->mWindow;
>> (gdb) p pVAAPICNTX->vaDisplay
>> $3 = (void *) 0xffffffffe42d3160
>>
>>
>> pVAAPICNTX is malloced struct ptr and one of the element of it is the
>> vaDisplay declated like this :
>>
>> typedef struct _VAAPICNTX
>> {
>>     uint32_t                  VAContextID;
>>    Window                   voWindow;
>>    int voWidth;
>>    int voHeight;
>> ;;;
>> ;;;
>>     VADisplay               vaDisplay;
>>
>> ;;;
>>
>> ;;;
>>
>> }VAAPICNTX, *PVAAPICNTX;
>>
>>
>> As you can see, dpy above around line 195 in  vaGetDisplay function is
>> shown as 0x7fffe42d3160 but once the program counter comes out of that
>> function and stack rolling happens, and context switches to upper level
>> code, the assigned value has  0xffffffffe42d3160 (sign extended vs
>> unsigned). What might cause this ? Next instance when I call vaInitialize
>> with this vaDisplay, it craches:
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> vaDisplayIsValid (dpy=0xffffffffe42d3450) at va.c:102
>> 102        return pDisplayContext && (pDisplayContext->vadpy_magic ==
>> VA_DISPLAY_MAGIC) && pDisplayContext->vaIsValid(pDisplayContext);
>>
>>
>>
>> Any help/pointer would be appreciated. Thanks
>>
>>
>>
>> gcc --version
>> gcc (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4
>> Copyright (C) 2013 Free Software Foundation, Inc.
>> This is free software; see the source for copying conditions.  There is NO
>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>> PURPOSE.
>>
>> ls -la libc-2.19.so
>> -rwxr-xr-x 1 root root 1845024 Dec  4  2014 libc-2.19.so
>>
>> ldd --version
>> ldd (Ubuntu EGLIBC 2.19-0ubuntu6.5) 2.19
>>
>>
>> Platform:
>>
>> Ratin
>>
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/libva/attachments/20151021/15857c88/attachment.html>