In the last few months Chad Versace and I have put a lot of effort into reworking the core of Piglit with an eye towards being able to effectively test GLES, Wayland, Android, and core profiles. A lot of that work is still in progress, and since others are starting to join in this effort, it seems like a good time to lay out our long term plans so that we can coordinate. Once we've had a chance to discuss these plans on the mailing list and come to a consensus, we're hoping to upload the revised description to the Piglit website (<a href="http://piglit.freedesktop.org/">http://piglit.freedesktop.org/</a>) so that it's easy to find and refer to. Background: As of roughly January 2012, Piglit was almost exclusively geared toward testing desktop OpenGL versions 1.0 through 3.0, running on Linux, Mac OSX, and Windows. It used GLEW (<a href="http://glew.sourceforge.net/">http://glew.sourceforge.net/</a>) to dispatch to OpenGL functions, and GLUT (<a href="http://www.opengl.org/resources/libraries/glut/">http://www.opengl.org/resources/libraries/glut/</a>) to interact with the window system. Neither GLEW nor GLUT are compatible with GLES, Wayland, Android, or core profiles. Only a few GLES tests existed, but they were compiled separately from the GL tests, and they were not integrated tightly into the framework. Furthermore, tests that in theory should have been able to be run on both GL and GLES were only compiled for GL. Long term plan (and status): (1) Stop using GLEW, and instead use a custom dispatch mechanism (piglit-dispatch) which can be extended to support GLES and core profiles. Piglit-dispatch uses a set of function pointers (passed to it at initialization time) to determine how to late-bind to a given GL function--this will allow its behaviour to be altered based on run-time parameters (see items (3) and (4)). Piglit-dispatch consists mostly of generated code that is built from the gl.spec and enumext.spec files published on the OpenGL web site. These files will be extended as necessary to describe the differences between GL and GLES. Status: the dispatch mechanism has been written and deployed. Piglit no longer uses GLEW. However, the GLES-specific modifications to the .spec files have not yet been made, and the dispatch mechanism does not yet support GLES or core profiles. (2) Stop using GLUT, and instead use a new mechanism (Waffle) to create GL contexts and open windows. Status: an initial implementation of waffle has been written and deployed. Piglit can be compiled to use either Waffle or GLUT, based on a compile time switch. Not all desktop GL tests work properly with Waffle yet. All GLES tests work properly with Waffle, however, and GLES tests may only be built if the Waffle compilation option is chosen. (3) Stop building GLES tests separately from the GL tests. Instead, build a single set of test binaries that can test the GL API, the GLES 1 API, the GLES 2 API, or (if applicable) all three sequentially. If libraries for only a subset of these APIs are present on the system, the tests will automatically only exercise those APIs that are present. This will be possible because Waffle and piglit-dispatch use dlsym() (or its Windows equivalent) to late-bind to the GL and GLES functions. Status: GLES and GL tests are still built separately. Waffle is capable of selecting which API to use at runtime, but piglit-dispatch and the rest of piglit are not. (4) Similarly, we will build a single set of test binaries that can test X windows, Android, Wayland, gbm, or (if applicable) all of them sequentially, depending on command-line options. Windows and Mac OS tests will still be built separately of course, because those platforms aren't binary-compatible with each other or with Linux. Status: When Piglit is built with waffle support enabled, a single set of binaries can test X windows or Wayland depending on an environment variable. Testing on Android and gbm is not implemented yet. Testing on multiple window systems sequentially is not supported yet. (5) In order for the framework to know which window systems, GL APIs, and extensions are relevant to any particular test, we will add a declarative mechanism for each test to specify the test requirements. The declarative syntax will use macros or inline functions. For example, if a test should run on desktop GL implementations that support GL_EXT_framebuffer_object, desktop GL implementations version 3.0 or later, and on ES2 implementations, the test would contain a line that looks something like "test_params.requirements = OR3(AND(GL_API, EXTENSION("GL_EXT_framebuffer_object")), GL_API_VERSION(3, 0), GLES2_API)". This will replace the imperative mechanism that we currently use to limit a test to certain GL versions and extensions (the piglit_require_xxx() functions). Status: not yet started. (6) Since standard C doesn't allow functions to be called inside a global initializer, to facilitate (5) we will replace the main() function in tests/util/piglit-framework.c with a piglit_run() function which takes as its input a data structure describing the test requirements (as well as other initialization info like the initial window size). Each individual test will contain a main() function that populates the data structure and then calls piglit_run(). To avoid having to write a lot of boilerplate code in each individual test, we'll provide a macro that creates a standard main() function for those tests that don't need a lot of customization. Status: not yet started. (7) In order to allow the majority of piglit tests to run with core profiles, forward-compatible contexts, and GLES (which don't support depreceated functionality), we will write new wrapper functions in tests/util which provide low-level testing operations using whatever functionality is present. For example, we might have a function which replaces any references to "glVertex" in a GLSL program with references to a custom vertex attribute, for those implementations that don't support the deprecated "glVertex" feature. Status: a few wrapper functions already exist, but many more need to be written. (8) In order to allow a single test invocation to run in a number of configurations sequentially (e.g. to test a certain piece of functionality using both Wayland and X windows), we will extend the protocol used by tests to report results so that a single invocation of a test can report multiple test results. Status: not yet started. Implementation Strategy and immediate plans: The above description doesn't address what order we plan on implementing these features. We would like to stay flexible on the specific implementation order, so that we can adapt to unexpected dependencies and to allow people to contribute to the effort as their time becomes available. However, we would like to avoid regressions as much as possible, since a regression in piglit is likely to be misinterpreted as a regression in Mesa. Here is what we expect to be working on soon: (a) Replace the main() function in tests/util/piglit-framework.c with individual main() functions in each test (see (6) above). (b) Add the necessary information to gl.spec and enumext.spec to indicate which functions are avaiable in GLES, and to describe GLES extensions (see (1) above). (c) Modify piglit-dispatch code generation scripts to consume the information from (b) and dispatch properly to GLES. (d) Add Windows and MacOS X support to Waffle. (e) Add basic keyboard input support to Waffle. This is necessary because most tests recognize "ESC" as a command to exit the test (if not run in "-auto" mode), and because a few tests recognize other keys (e.g. fbo tests that use keyboard input to cycle between framebuffer formats). (f) Remove the compile-time switch that enables Waffle, and switch over to using Waffle for all desktop GL tests. Note: since we want to keep Piglit viable as a testing platform for Mesa while these tasks are in progress, we can't do this until after doing (d) and (e). However, we have already switched over to exclusively using Waffle for GLES testing.