[Libva] Weird memory problem around vaGetDisplay

[Emil Velikov]

Hi Ratin,

I would strongly recommend going through the warning and fixing the
(your and/or libva) code. I realise how tedious this might be,
although it will save you a ton of time/headaches in the long run.
Same goes for -Wextra ;-)

Regards,
Emil

On 23 October 2015 at 19:15, Ratin <ratin3 at gmail.com> wrote:
> Hi Emil, I only had basic warnings enabled (-Wpedantic -Wformat etc) since I
> get tons of non-severe warnings. I added -Wall and I do get "implicit
> declaration of function 'vaGetDisplay' [-Wimplicit-function-declaration]". I
> will keep this enabled and perhaps redirect compiler output to a file and
> grep for serious warnings.
>
> Thanks
>
> Ratin
>
>
>
>
>
>
>
> On Thu, Oct 22, 2015 at 1:49 AM, Emil Velikov <emil.l.velikov at gmail.com>
> wrote:
>>
>> Hi Ratin,
>>
>> Out of curiosity - when you say libva_x11.h you meant
>> /usr/include/va/va_x11.h, is that correct ? Upon closer look, the
>> header seems to be the only place where the function is declared, thus
>> the compiler should have thrown a warning. Do you have those disabled
>> by any chance ?
>>
>> Cheers,
>> Emil
>>
>>
>> On 22 October 2015 at 01:47, Ratin <ratin3 at gmail.com> wrote:
>> > Found the culprit, it was missing header (libva_x11.h). If anybody
>> > interested to know, this caused the compiler to think the function will
>> > return a 32 bit pointer, so the compiler inserted a cltq statement into
>> > the
>> > assembly code which was flipping the upper 32 bits to all 1s.
>> >
>> > On Wed, Oct 21, 2015 at 12:15 PM, Ratin <ratin3 at gmail.com> wrote:
>> >>
>> >> # uname -a
>> >> Linux nuc1 3.16.0-51-generic #69~14.04.1-Ubuntu SMP Wed Oct 7 15:32:41
>> >> UTC
>> >> 2015 x86_64 x86_64 x86_64 GNU/Linux
>> >>
>> >>
>> >>
>> >>
>> >> On Wed, Oct 21, 2015 at 12:13 PM, Ratin <ratin3 at gmail.com> wrote:
>> >>>
>> >>> Breakpoint 1, vaGetDisplay (native_dpy=0x7fffe4312870) at va_x11.c:154
>> >>> 154    {
>> >>> (gdb) list
>> >>> 149
>> >>> 150
>> >>> 151    VADisplay vaGetDisplay (
>> >>> 152        Display *native_dpy /* implementation specific */
>> >>> 153    )
>> >>> 154    {
>> >>> 155      VADisplay dpy = NULL;
>> >>> 156      VADisplayContextP pDisplayContext;
>> >>> 157
>> >>> ;;;
>> >>> 179          pDisplayContext->opaque          = NULL;
>> >>> 177          pDisplayContext->vaDestroy       =
>> >>> va_DisplayContextDestroy;
>> >>> 178          pDisplayContext->vaGetDriverName =
>> >>> va_DisplayContextGetDriverName;
>> >>> 181          dpy                              =
>> >>> (VADisplay)pDisplayContext;
>> >>> 195    }
>> >>> (gdb) p dpy
>> >>> $2 = (VADisplay) 0x7fffe42d3160
>> >>> (gdb) n <========================== stack unroll happens here
>> >>> (gdb) list
>> >>> 935            pVAAPICNTX->vaDisplay = (VADisplay)  <==== upper level
>> >>> code vaGetDisplay(pVAAPICNTX->mX11Display);
>> >>> 936             pVAAPICNTX->voWindow = pRendererParams->mWindow;
>> >>> (gdb) p pVAAPICNTX->vaDisplay
>> >>> $3 = (void *) 0xffffffffe42d3160
>> >>>
>> >>>
>> >>> pVAAPICNTX is malloced struct ptr and one of the element of it is the
>> >>> vaDisplay declated like this :
>> >>>
>> >>> typedef struct _VAAPICNTX
>> >>> {
>> >>>     uint32_t                  VAContextID;
>> >>>    Window                   voWindow;
>> >>>    int voWidth;
>> >>>    int voHeight;
>> >>> ;;;
>> >>> ;;;
>> >>>     VADisplay               vaDisplay;
>> >>>
>> >>> ;;;
>> >>>
>> >>> ;;;
>> >>>
>> >>> }VAAPICNTX, *PVAAPICNTX;
>> >>>
>> >>>
>> >>> As you can see, dpy above around line 195 in  vaGetDisplay function is
>> >>> shown as 0x7fffe42d3160 but once the program counter comes out of that
>> >>> function and stack rolling happens, and context switches to upper
>> >>> level
>> >>> code, the assigned value has  0xffffffffe42d3160 (sign extended vs
>> >>> unsigned). What might cause this ? Next instance when I call
>> >>> vaInitialize
>> >>> with this vaDisplay, it craches:
>> >>>
>> >>> Program received signal SIGSEGV, Segmentation fault.
>> >>> vaDisplayIsValid (dpy=0xffffffffe42d3450) at va.c:102
>> >>> 102        return pDisplayContext && (pDisplayContext->vadpy_magic ==
>> >>> VA_DISPLAY_MAGIC) && pDisplayContext->vaIsValid(pDisplayContext);
>> >>>
>> >>>
>> >>>
>> >>> Any help/pointer would be appreciated. Thanks
>> >>>
>> >>>
>> >>>
>> >>> gcc --version
>> >>> gcc (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4
>> >>> Copyright (C) 2013 Free Software Foundation, Inc.
>> >>> This is free software; see the source for copying conditions.  There
>> >>> is
>> >>> NO
>> >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>> >>> PURPOSE.
>> >>>
>> >>> ls -la libc-2.19.so
>> >>> -rwxr-xr-x 1 root root 1845024 Dec  4  2014 libc-2.19.so
>> >>>
>> >>> ldd --version
>> >>> ldd (Ubuntu EGLIBC 2.19-0ubuntu6.5) 2.19
>> >>>
>> >>>
>> >>> Platform:
>> >>>
>> >>> Ratin
>> >>>
>> >>>
>> >>>
>> >>>
>> >>
>> >
>> >
>> > _______________________________________________
>> > Libva mailing list
>> > Libva at lists.freedesktop.org
>> > http://lists.freedesktop.org/mailman/listinfo/libva
>> >