[Libva] FGLRX detection code overly restrictive

[Stephan Diestelhorst]

Hi,
 I have tried to get VA-API's XvBA backend working on my Zacate system
and eventually succeeded. But first I was confronted with segfaults from all
applications I tried (VLC, mplayer, hwdecode-demos), with the following
backtrace:

__strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:32caused by

__strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:32

32      ../sysdeps/x86_64/multiarch/../strlen.S: No such file or directory.
       in ../sysdeps/x86_64/multiarch/../strlen.S
(gdb) bt
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:32
#1  0x00007ffff6af511e in va_openDriver (dpy=0x623a30,
major_version=0x7fffffffe00c, minor_version=0x7fffffffe008) at
va.c:180
#2  vaInitialize (dpy=0x623a30, major_version=0x7fffffffe00c,
minor_version=0x7fffffffe008) at va.c:388
#3  0x000000000040b18c in vaapi_init (display=0x623a30) at vaapi.c:137
#4  0x00000000004051b2 in main (argc=<value optimized out>,
argv=<value optimized out>) at common.c:1060

This is the first bug: handing in a NULL pointer to strlen without checking.

Now some digging shows that somehow fglrx does not get detected here, although
present. The curious thing is that vainfo, which uses the same
vaInitialize call, does
not suffer from this. Turns out that vainfo is the only application
that gets a specifc
display on ":0.0" through XOpenDisplay, while all other applications
pass whatever
display they currently use to the init call.

Eventually, this causes a mismatch in match_display in
libva/va/x11/va_fglrx.c such
that x11_dpy_name = ":0", test_dpy_name=":0.0" (and display_name being
":0" in there)
which causes the comparison to fail and hence does not set the driver
name to fglrx,
but returns a success error code.

Now, I do not know anything about the deeper reasoning behind all
this, but it seems
that this is inconsistent.

The workaround is of course: export LIBVA_DRIVER_NAME=fglrx which is ugly.

Stephan