[Libva] Weird memory problem around vaGetDisplay

[Emil Velikov]

Hi Ratin,

Out of curiosity - when you say libva_x11.h you meant
/usr/include/va/va_x11.h, is that correct ? Upon closer look, the
header seems to be the only place where the function is declared, thus
the compiler should have thrown a warning. Do you have those disabled
by any chance ?

Cheers,
Emil


On 22 October 2015 at 01:47, Ratin <ratin3 at gmail.com> wrote:
> Found the culprit, it was missing header (libva_x11.h). If anybody
> interested to know, this caused the compiler to think the function will
> return a 32 bit pointer, so the compiler inserted a cltq statement into the
> assembly code which was flipping the upper 32 bits to all 1s.
>
> On Wed, Oct 21, 2015 at 12:15 PM, Ratin <ratin3 at gmail.com> wrote:
>>
>> # uname -a
>> Linux nuc1 3.16.0-51-generic #69~14.04.1-Ubuntu SMP Wed Oct 7 15:32:41 UTC
>> 2015 x86_64 x86_64 x86_64 GNU/Linux
>>
>>
>>
>>
>> On Wed, Oct 21, 2015 at 12:13 PM, Ratin <ratin3 at gmail.com> wrote:
>>>
>>> Breakpoint 1, vaGetDisplay (native_dpy=0x7fffe4312870) at va_x11.c:154
>>> 154    {
>>> (gdb) list
>>> 149
>>> 150
>>> 151    VADisplay vaGetDisplay (
>>> 152        Display *native_dpy /* implementation specific */
>>> 153    )
>>> 154    {
>>> 155      VADisplay dpy = NULL;
>>> 156      VADisplayContextP pDisplayContext;
>>> 157
>>> ;;;
>>> 179          pDisplayContext->opaque          = NULL;
>>> 177          pDisplayContext->vaDestroy       = va_DisplayContextDestroy;
>>> 178          pDisplayContext->vaGetDriverName =
>>> va_DisplayContextGetDriverName;
>>> 181          dpy                              =
>>> (VADisplay)pDisplayContext;
>>> 195    }
>>> (gdb) p dpy
>>> $2 = (VADisplay) 0x7fffe42d3160
>>> (gdb) n <========================== stack unroll happens here
>>> (gdb) list
>>> 935            pVAAPICNTX->vaDisplay = (VADisplay)  <==== upper level
>>> code vaGetDisplay(pVAAPICNTX->mX11Display);
>>> 936             pVAAPICNTX->voWindow = pRendererParams->mWindow;
>>> (gdb) p pVAAPICNTX->vaDisplay
>>> $3 = (void *) 0xffffffffe42d3160
>>>
>>>
>>> pVAAPICNTX is malloced struct ptr and one of the element of it is the
>>> vaDisplay declated like this :
>>>
>>> typedef struct _VAAPICNTX
>>> {
>>>     uint32_t                  VAContextID;
>>>    Window                   voWindow;
>>>    int voWidth;
>>>    int voHeight;
>>> ;;;
>>> ;;;
>>>     VADisplay               vaDisplay;
>>>
>>> ;;;
>>>
>>> ;;;
>>>
>>> }VAAPICNTX, *PVAAPICNTX;
>>>
>>>
>>> As you can see, dpy above around line 195 in  vaGetDisplay function is
>>> shown as 0x7fffe42d3160 but once the program counter comes out of that
>>> function and stack rolling happens, and context switches to upper level
>>> code, the assigned value has  0xffffffffe42d3160 (sign extended vs
>>> unsigned). What might cause this ? Next instance when I call vaInitialize
>>> with this vaDisplay, it craches:
>>>
>>> Program received signal SIGSEGV, Segmentation fault.
>>> vaDisplayIsValid (dpy=0xffffffffe42d3450) at va.c:102
>>> 102        return pDisplayContext && (pDisplayContext->vadpy_magic ==
>>> VA_DISPLAY_MAGIC) && pDisplayContext->vaIsValid(pDisplayContext);
>>>
>>>
>>>
>>> Any help/pointer would be appreciated. Thanks
>>>
>>>
>>>
>>> gcc --version
>>> gcc (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4
>>> Copyright (C) 2013 Free Software Foundation, Inc.
>>> This is free software; see the source for copying conditions.  There is
>>> NO
>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>>> PURPOSE.
>>>
>>> ls -la libc-2.19.so
>>> -rwxr-xr-x 1 root root 1845024 Dec  4  2014 libc-2.19.so
>>>
>>> ldd --version
>>> ldd (Ubuntu EGLIBC 2.19-0ubuntu6.5) 2.19
>>>
>>>
>>> Platform:
>>>
>>> Ratin
>>>
>>>
>>>
>>>
>>
>
>
> _______________________________________________
> Libva mailing list
> Libva at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/libva
>