[Piglit] Waffle and piglit-dispatch plans

[Paul Berry]

In the last few months Chad Versace and I have put a lot of effort
into reworking the core of Piglit with an eye towards being able to
effectively test GLES, Wayland, Android, and core profiles.  A lot of
that work is still in progress, and since others are starting to join
in this effort, it seems like a good time to lay out our long term
plans so that we can coordinate.

Once we've had a chance to discuss these plans on the mailing list and
come to a consensus, we're hoping to upload the revised description to
the Piglit website (http://piglit.freedesktop.org/) so that it's easy
to find and refer to.


Background:

As of roughly January 2012, Piglit was almost exclusively geared
toward testing desktop OpenGL versions 1.0 through 3.0, running on
Linux, Mac OSX, and Windows.  It used GLEW
(http://glew.sourceforge.net/) to dispatch to OpenGL functions, and
GLUT (http://www.opengl.org/resources/libraries/glut/) to interact
with the window system.  Neither GLEW nor GLUT are compatible with
GLES, Wayland, Android, or core profiles.

Only a few GLES tests existed, but they were compiled separately from
the GL tests, and they were not integrated tightly into the framework.
Furthermore, tests that in theory should have been able to be run on
both GL and GLES were only compiled for GL.


Long term plan (and status):

(1) Stop using GLEW, and instead use a custom dispatch mechanism
(piglit-dispatch) which can be extended to support GLES and core
profiles.  Piglit-dispatch uses a set of function pointers (passed to
it at initialization time) to determine how to late-bind to a given GL
function--this will allow its behaviour to be altered based on
run-time parameters (see items (3) and (4)).  Piglit-dispatch consists
mostly of generated code that is built from the gl.spec and
enumext.spec files published on the OpenGL web site.  These files will
be extended as necessary to describe the differences between GL and
GLES.  Status: the dispatch mechanism has been written and deployed.
Piglit no longer uses GLEW.  However, the GLES-specific modifications
to the .spec files have not yet been made, and the dispatch mechanism
does not yet support GLES or core profiles.

(2) Stop using GLUT, and instead use a new mechanism (Waffle) to
create GL contexts and open windows.  Status: an initial
implementation of waffle has been written and deployed.  Piglit can be
compiled to use either Waffle or GLUT, based on a compile time switch.
Not all desktop GL tests work properly with Waffle yet.  All GLES
tests work properly with Waffle, however, and GLES tests may only be
built if the Waffle compilation option is chosen.

(3) Stop building GLES tests separately from the GL tests.  Instead,
build a single set of test binaries that can test the GL API, the GLES
1 API, the GLES 2 API, or (if applicable) all three sequentially.  If
libraries for only a subset of these APIs are present on the system,
the tests will automatically only exercise those APIs that are
present.  This will be possible because Waffle and piglit-dispatch use
dlsym() (or its Windows equivalent) to late-bind to the GL and GLES
functions.  Status: GLES and GL tests are still built separately.
Waffle is capable of selecting which API to use at runtime, but
piglit-dispatch and the rest of piglit are not.

(4) Similarly, we will build a single set of test binaries that can
test X windows, Android, Wayland, gbm, or (if applicable) all of them
sequentially, depending on command-line options.  Windows and Mac OS
tests will still be built separately of course, because those
platforms aren't binary-compatible with each other or with Linux.
Status: When Piglit is built with waffle support enabled, a single set
of binaries can test X windows or Wayland depending on an environment
variable.  Testing on Android and gbm is not implemented yet.  Testing
on multiple window systems sequentially is not supported yet.

(5) In order for the framework to know which window systems, GL APIs,
and extensions are relevant to any particular test, we will add a
declarative mechanism for each test to specify the test requirements.
The declarative syntax will use macros or inline functions.  For
example, if a test should run on desktop GL implementations that
support GL_EXT_framebuffer_object, desktop GL implementations version
3.0 or later, and on ES2 implementations, the test would contain a
line that looks something like "test_params.requirements =
OR3(AND(GL_API, EXTENSION("GL_EXT_framebuffer_object")),
GL_API_VERSION(3, 0), GLES2_API)".  This will replace the imperative
mechanism that we currently use to limit a test to certain GL versions
and extensions (the piglit_require_xxx() functions).  Status: not yet
started.

(6) Since standard C doesn't allow functions to be called inside a
global initializer, to facilitate (5) we will replace the main()
function in tests/util/piglit-framework.c with a piglit_run() function
which takes as its input a data structure describing the test
requirements (as well as other initialization info like the initial
window size).  Each individual test will contain a main() function
that populates the data structure and then calls piglit_run().  To
avoid having to write a lot of boilerplate code in each individual
test, we'll provide a macro that creates a standard main() function
for those tests that don't need a lot of customization.  Status: not
yet started.

(7) In order to allow the majority of piglit tests to run with core
profiles, forward-compatible contexts, and GLES (which don't support
depreceated functionality), we will write new wrapper functions in
tests/util which provide low-level testing operations using whatever
functionality is present.  For example, we might have a function which
replaces any references to "glVertex" in a GLSL program with
references to a custom vertex attribute, for those implementations
that don't support the deprecated "glVertex" feature.  Status: a few
wrapper functions already exist, but many more need to be written.

(8) In order to allow a single test invocation to run in a number of
configurations sequentially (e.g. to test a certain piece of
functionality using both Wayland and X windows), we will extend the
protocol used by tests to report results so that a single invocation
of a test can report multiple test results.  Status: not yet started.


Implementation Strategy and immediate plans:

The above description doesn't address what order we plan on
implementing these features.  We would like to stay flexible on the
specific implementation order, so that we can adapt to unexpected
dependencies and to allow people to contribute to the effort as their
time becomes available.  However, we would like to avoid regressions
as much as possible, since a regression in piglit is likely to be
misinterpreted as a regression in Mesa.

Here is what we expect to be working on soon:

(a) Replace the main() function in tests/util/piglit-framework.c with
individual main() functions in each test (see (6) above).

(b) Add the necessary information to gl.spec and enumext.spec to
indicate which functions are avaiable in GLES, and to describe GLES
extensions (see (1) above).

(c) Modify piglit-dispatch code generation scripts to consume the
information from (b) and dispatch properly to GLES.

(d) Add Windows and MacOS X support to Waffle.

(e) Add basic keyboard input support to Waffle.  This is necessary
because most tests recognize "ESC" as a command to exit the test (if
not run in "-auto" mode), and because a few tests recognize other keys
(e.g. fbo tests that use keyboard input to cycle between framebuffer
formats).

(f) Remove the compile-time switch that enables Waffle, and switch
over to using Waffle for all desktop GL tests.  Note: since we want to
keep Piglit viable as a testing platform for Mesa while these tasks
are in progress, we can't do this until after doing (d) and (e).
However, we have already switched over to exclusively using Waffle for
GLES testing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/piglit/attachments/20120525/b4ab31e4/attachment.htm>