[Mesa-dev] WebGL test suite, and whitelisting drivers / OpenGL implementations in Firefox

[Benoit Jacob]

Hi List,

I realize that you may well not have time to care about WebGL [1] or 
Firefox. In that case, I'm still writing because there's a new test 
suite, namely the WebGL conformance test suite, that may be of interest 
to help find and fix bugs in OpenGL implementations.

The goal of this email is to discuss steps towards whitelisting Xorg 
OpenGL drivers for WebGL rendering, and more generally for all 
OpenGL-based features, in Firefox. Although I'm only directly concerned 
with Firefox, this really applies equally well to all browsers 
implementing WebGL.

OpenGL-based features are disabled by default in Firefox 4 on X11 
because of OpenGL bugs we've hit [2], and currently only the NVIDIA 
proprietary driver is whitelisted [3].

The good news is that there's a good official WebGL conformance test 
suite, and that any driver passing (most of) it will be a great 
candidate for whitelisting.

To run it and reproduce the issues we're having, you can follow these steps:

1. Download a nightly build of Firefox 4 there, and untar it:

    http://nightly.mozilla.org/

2. Run it with the MOZ_GLX_IGNORE_BLACKLIST env variable, e.g.:

    $ MOZ_GLX_IGNORE_BLACKLIST=1 ./firefox -P -no-remote

    (-P allows you to create a fresh new profile and -no-remote
    prevents attaching to existing firefox instance).

3. Go to this URL:

 
https://cvs.khronos.org/svn/repos/registry/trunk/public/webgl/sdk/tests/webgl-conformance-tests.html

4. Click 'run tests'. If it doesn't crash and more than 90% tests 
succeed, we can start discussing whitelisting, depending on what the 
failures are (see below). Notice that this runs two sets of tests, 
conformance/ and more/. Only conformance/ really matters. Also, a few 
failures will be Firefox bugs (the WebGL conformance test suite is still 
somewhat of a moving target) and some other failures may be bugs in the 
tests themselves (this applies mostly to the tests under more/). To give 
you an idea of what to aim for, here on debian sid x86-64 + NVIDIA 
proprietary driver I get this:

    Results: (5241 of 5315 passed, 3 timed out)

And among the conformance/ test pages, I get these failures on that system:
   conformance/context-lost-restored.html(*timeout*)
   conformance/context-lost.html(*timeout*)
   conformance/tex-image-and-sub-image-2d-with-video.html(*timeout*)
   conformance/array-unit-tests.html (278 of 279 passed)
   conformance/gl-get-active-attribute.html (19 of 22 passed)
   conformance/gl-get-calls.html (73 of 75 passed)
   conformance/gl-getshadersource.html (1 of 3 passed)
   conformance/gl-uniform-bool.html (1 of 2 passed)
   conformance/glsl-conformance.html (101 of 104 passed)
   conformance/invalid-passed-params.html (50 of 74 passed)
   conformance/is-object.html (24 of 25 passed)
   conformance/read-pixels-test.html (121 of 125 passed)
   conformance/uninitialized-test.html (11 of 19 passed)
   conformance/webgl-specific.html (31 of 40 passed)


5. A few Firefox debugging infos/tips that you may find useful:
   - when using OSMesa, the whole WebGL conformance test suite runs 
without any legitimate valgrind errors [4].
   - defining the MOZ_X_SYNC env variable enables XSynchronize
   - when using a debug build of Firefox, defining MOZ_GL_DEBUG enables 
a special "openGL debugging mode" that calls glFinish() after every GL 
call, that checks that then OpenGL context that a GL call is supposed to 
apply to is indeed the current context (or else it aborts), and that 
logs GL errors. MOZ_GL_DEBUG_VERBOSE will log every GL call.
   - debug builds of Firefox may be downloaded from
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-debug/
but they don't have debug symbols AFAIK. For that, you may need to make 
your own build [5].
   - to run firefox in GDB, do firefox -g. When attaching GDB to already 
running firefox, attach to firefox-bin.


Cheers,
Benoit

Links:

[1] 
https://cvs.khronos.org/svn/repos/registry/trunk/public/webgl/doc/spec/WebGL-spec.html

[2] See for example:
       https://bugzilla.mozilla.org/show_bA ug.cgi?id=621699
       https://bugzilla.mozilla.org/show_bug.cgi?id=589546
       https://bugzilla.mozilla.org/show_bug.cgi?id=616416
       https://bugs.freedesktop.org/show_bug.cgi?id=32238
     Also you may find this useful:
       http://crash-stats.mozilla.com/query#
     Sample query for all crashes in r600_dri.so in Firefox 4 ("branch 
2.0") that are NOT caused by a plugin like Flash during the past week 
(the low volume of crashes is explained by the fact that we're not 
whitelisting this driver so only the few people running with 
MOZ_GLX_IGNORE_BLACKLIST are exposed):
 
http://crash-stats.mozilla.com/query/query?product=Firefox&version=ALL%3AALL&platform=linux&branch=2.0&range_value=1&range_unit=weeks&date=01%2F16%2F2011+18%3A05%3A19&query_search=signature&query_type=contains&query=r600_dri.so&build_id=&process_type=browser&hang_type=any&do_query=1

[3] The whitelisting code is currently there:
 
http://hg.mozilla.org/mozilla-central/file/f9f48079910f/gfx/thebes/GLContextProviderGLX.cpp#l238

[4] See:
     https://bugzilla.mozilla.org/show_bug.cgi?id=588918
To anyone wanting to reproduce this, you need a recent Mesa (post bug 
31837), a build of firefox with --enable-valgrind --disable-jemalloc, 
and the valgrind suppressions file given on bug 588918.

[5] See:
     https://developer.mozilla.org/En/Simple_Firefox_build