[Libva] Weird memory problem around vaGetDisplay

[Ratin]

Hi Emil, I only had basic warnings enabled (-Wpedantic -Wformat etc) since
I get tons of non-severe warnings. I added -Wall and I do get "implicit
declaration of function 'vaGetDisplay' [-Wimplicit-function-declaration]".
I will keep this enabled and perhaps redirect compiler output to a file and
grep for serious warnings.

Thanks

Ratin







On Thu, Oct 22, 2015 at 1:49 AM, Emil Velikov <emil.l.velikov at gmail.com>
wrote:

> Hi Ratin,
>
> Out of curiosity - when you say libva_x11.h you meant
> /usr/include/va/va_x11.h, is that correct ? Upon closer look, the
> header seems to be the only place where the function is declared, thus
> the compiler should have thrown a warning. Do you have those disabled
> by any chance ?
>
> Cheers,
> Emil
>
>
> On 22 October 2015 at 01:47, Ratin <ratin3 at gmail.com> wrote:
> > Found the culprit, it was missing header (libva_x11.h). If anybody
> > interested to know, this caused the compiler to think the function will
> > return a 32 bit pointer, so the compiler inserted a cltq statement into
> the
> > assembly code which was flipping the upper 32 bits to all 1s.
> >
> > On Wed, Oct 21, 2015 at 12:15 PM, Ratin <ratin3 at gmail.com> wrote:
> >>
> >> # uname -a
> >> Linux nuc1 3.16.0-51-generic #69~14.04.1-Ubuntu SMP Wed Oct 7 15:32:41
> UTC
> >> 2015 x86_64 x86_64 x86_64 GNU/Linux
> >>
> >>
> >>
> >>
> >> On Wed, Oct 21, 2015 at 12:13 PM, Ratin <ratin3 at gmail.com> wrote:
> >>>
> >>> Breakpoint 1, vaGetDisplay (native_dpy=0x7fffe4312870) at va_x11.c:154
> >>> 154    {
> >>> (gdb) list
> >>> 149
> >>> 150
> >>> 151    VADisplay vaGetDisplay (
> >>> 152        Display *native_dpy /* implementation specific */
> >>> 153    )
> >>> 154    {
> >>> 155      VADisplay dpy = NULL;
> >>> 156      VADisplayContextP pDisplayContext;
> >>> 157
> >>> ;;;
> >>> 179          pDisplayContext->opaque          = NULL;
> >>> 177          pDisplayContext->vaDestroy       =
> va_DisplayContextDestroy;
> >>> 178          pDisplayContext->vaGetDriverName =
> >>> va_DisplayContextGetDriverName;
> >>> 181          dpy                              =
> >>> (VADisplay)pDisplayContext;
> >>> 195    }
> >>> (gdb) p dpy
> >>> $2 = (VADisplay) 0x7fffe42d3160
> >>> (gdb) n <========================== stack unroll happens here
> >>> (gdb) list
> >>> 935            pVAAPICNTX->vaDisplay = (VADisplay)  <==== upper level
> >>> code vaGetDisplay(pVAAPICNTX->mX11Display);
> >>> 936             pVAAPICNTX->voWindow = pRendererParams->mWindow;
> >>> (gdb) p pVAAPICNTX->vaDisplay
> >>> $3 = (void *) 0xffffffffe42d3160
> >>>
> >>>
> >>> pVAAPICNTX is malloced struct ptr and one of the element of it is the
> >>> vaDisplay declated like this :
> >>>
> >>> typedef struct _VAAPICNTX
> >>> {
> >>>     uint32_t                  VAContextID;
> >>>    Window                   voWindow;
> >>>    int voWidth;
> >>>    int voHeight;
> >>> ;;;
> >>> ;;;
> >>>     VADisplay               vaDisplay;
> >>>
> >>> ;;;
> >>>
> >>> ;;;
> >>>
> >>> }VAAPICNTX, *PVAAPICNTX;
> >>>
> >>>
> >>> As you can see, dpy above around line 195 in  vaGetDisplay function is
> >>> shown as 0x7fffe42d3160 but once the program counter comes out of that
> >>> function and stack rolling happens, and context switches to upper level
> >>> code, the assigned value has  0xffffffffe42d3160 (sign extended vs
> >>> unsigned). What might cause this ? Next instance when I call
> vaInitialize
> >>> with this vaDisplay, it craches:
> >>>
> >>> Program received signal SIGSEGV, Segmentation fault.
> >>> vaDisplayIsValid (dpy=0xffffffffe42d3450) at va.c:102
> >>> 102        return pDisplayContext && (pDisplayContext->vadpy_magic ==
> >>> VA_DISPLAY_MAGIC) && pDisplayContext->vaIsValid(pDisplayContext);
> >>>
> >>>
> >>>
> >>> Any help/pointer would be appreciated. Thanks
> >>>
> >>>
> >>>
> >>> gcc --version
> >>> gcc (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4
> >>> Copyright (C) 2013 Free Software Foundation, Inc.
> >>> This is free software; see the source for copying conditions.  There is
> >>> NO
> >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
> >>> PURPOSE.
> >>>
> >>> ls -la libc-2.19.so
> >>> -rwxr-xr-x 1 root root 1845024 Dec  4  2014 libc-2.19.so
> >>>
> >>> ldd --version
> >>> ldd (Ubuntu EGLIBC 2.19-0ubuntu6.5) 2.19
> >>>
> >>>
> >>> Platform:
> >>>
> >>> Ratin
> >>>
> >>>
> >>>
> >>>
> >>
> >
> >
> > _______________________________________________
> > Libva mailing list
> > Libva at lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/libva
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/libva/attachments/20151023/d1f918d8/attachment.html>