[Piglit] Waffle and piglit-dispatch plans

[Kenneth Graunke]

On 05/29/2012 01:47 PM, Pauli Nieminen wrote:
> On Tue, May 29, 2012 at 11:17:26AM -0700, Chad Versace wrote:
>> On 05/29/2012 06:30 AM, Pauli Nieminen wrote:
>>> On Fri, May 25, 2012 at 01:40:10PM -0700, Paul Berry wrote:
>>>>    In the last few months Chad Versace and I have put a lot of effort
>>>>    into reworking the core of Piglit with an eye towards being able to
>>>>    effectively test GLES, Wayland, Android, and core profiles.  A lot of
>>>>    that work is still in progress, and since others are starting to join
>>>>    in this effort, it seems like a good time to lay out our long term
>>>>    plans so that we can coordinate.
>>>>
>>>>    Once we've had a chance to discuss these plans on the mailing list and
>>>>    come to a consensus, we're hoping to upload the revised description to
>>>>    the Piglit website (http://piglit.freedesktop.org/) so that it's easy
>>>>    to find and refer to.
>>>>
>>>>    Background:
>>>>
>>>>    As of roughly January 2012, Piglit was almost exclusively geared
>>>>    toward testing desktop OpenGL versions 1.0 through 3.0, running on
>>>>    Linux, Mac OSX, and Windows.  It used GLEW
>>>>    (http://glew.sourceforge.net/) to dispatch to OpenGL functions, and
>>>>    GLUT (http://www.opengl.org/resources/libraries/glut/) to interact
>>>>    with the window system.  Neither GLEW nor GLUT are compatible with
>>>>    GLES, Wayland, Android, or core profiles.
>>>>
>>>>    Only a few GLES tests existed, but they were compiled separately from
>>>>    the GL tests, and they were not integrated tightly into the framework.
>>>>    Furthermore, tests that in theory should have been able to be run on
>>>>    both GL and GLES were only compiled for GL.
>>>>
>>>>    Long term plan (and status):
>>>>
>>>>    (5) In order for the framework to know which window systems, GL APIs,
>>>>    and extensions are relevant to any particular test, we will add a
>>>>    declarative mechanism for each test to specify the test requirements.
>>>>    The declarative syntax will use macros or inline functions.  For
>>>>    example, if a test should run on desktop GL implementations that
>>>>    support GL_EXT_framebuffer_object, desktop GL implementations version
>>>>    3.0 or later, and on ES2 implementations, the test would contain a
>>>>    line that looks something like "test_params.requirements =
>>>>    OR3(AND(GL_API, EXTENSION("GL_EXT_framebuffer_object")),
>>>>    GL_API_VERSION(3, 0), GLES2_API)".  This will replace the imperative
>>>>    mechanism that we currently use to limit a test to certain GL versions
>>>>    and extensions (the piglit_require_xxx() functions).  Status: not yet
>>>>    started.
>>>>
>>>
>>> This makes it only slightly easier that using piglit_is_supported
>>> functions. That was mostly reason why I added runtime checks for
>>> platform to GLX and EGL code.
>>>
>>> Of course it is nice to write common and possible complex requirements
>>> for tests in standard simple to write fashion. But most likely tests
>>> will still have to do runtime branching depending on if it is running
>>> on GLES or GL.
>>
>> Having the test requirements stated declaratively allows us to do several things
>> that are difficult, or not possible, with the piglit_is_supported functions. Again,
>> these things have been part of Ken's, Paul's, and my plan for some time,
>> but we forgot to document it.
>>
>> 1) Filtering tests by GL API.
>>
>> Eventually, a large portion of tests will be capable of running under GL and GLES.
>> Once we reach that point, developers will normally not want to run all those
>> tests twice, first as GL and then as GLES. The framework will need to be intelligent
>> enough to, by request, run only GL test cases, only GLES test cases, or run GL and GLES test
>> cases (in which case some tests will be ran multiple times).
>>
>> To allow this, the plan is to add some standard command-line option to test
>> executables and to the Piglit python scripts, such as --gl-api. If --gl-api=gles2
>> is given to a test executable, then it will run only its GLES2 test cases. Given
>> --gl-api=all, it will run test cases for all APIs. This plan is not yet set in
>> stone, and we're receptive to suggestions.
>>
>> Similarly, if --gl-api=gles2 is given to piglit-print-commands.py, then the script
>> will  1) invoke each test executable with a flag that requests it to print out info
>> about its requirements, and 2) filter out tests that do no support GLES2. This
>> will give developers fine grained control over which tests they run, control
>> that would be cumbersome to provide with the piglit-run.py's -x option. And this
>> gives QA the ability to filter out inappropriate test cases when running Piglit
>> on Android.
>>
>> 2) Filtering window system platforms.
>>
>> Once support for the --gl-api option is implemented, the next logical step
>> is to add a --platform={all,glx,wayland,x11_egl} option that behaves analogously.
>>
> 
> yes. Standard checks are good way. If I understand your proposal
> correctly tests binary is supposed to tell about if it is run able in
> given platform.
> 
> Wouldn't that makes it simpler to have same tests set in all
> platforms and GL flavors? Tests automatically skip when given
> environment doesn't match what test requires. Fork-exec-skip should be
> fast enough not to affect total runtime much.

It actually does take a while.  Running all the 3.0+/1.30+ tests on
pre-Sandybridge skips quite a few tests already...and more all the time.

Plus, we'd eventually like to include FBConfigs/visuals in the list of
requirements (i.e. RGBA, at least N bits of depth, stencil), and that
will create a -lot- of combinations.

The idea is to improve the runner infrastructure to select reasonable
combinations of tests rather than always running them all.