[Piglit] Waffle and piglit-dispatch plans

Wed May 30 16:06:47 PDT 2012

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/29/2012 01:47 PM, Pauli Nieminen wrote:
> On Tue, May 29, 2012 at 11:17:26AM -0700, Chad Versace wrote:
>> On 05/29/2012 06:30 AM, Pauli Nieminen wrote:
>>> On Fri, May 25, 2012 at 01:40:10PM -0700, Paul Berry wrote:
>>>>    In the last few months Chad Versace and I have put a lot of effort
>>>>    into reworking the core of Piglit with an eye towards being able to
>>>>    effectively test GLES, Wayland, Android, and core profiles.  A lot of
>>>>    that work is still in progress, and since others are starting to join
>>>>    in this effort, it seems like a good time to lay out our long term
>>>>    plans so that we can coordinate.
>>>>
>>>>    Once we've had a chance to discuss these plans on the mailing list and
>>>>    come to a consensus, we're hoping to upload the revised description to
>>>>    the Piglit website (http://piglit.freedesktop.org/) so that it's easy
>>>>    to find and refer to.
>>>>
>>>>    Background:
>>>>
>>>>    As of roughly January 2012, Piglit was almost exclusively geared
>>>>    toward testing desktop OpenGL versions 1.0 through 3.0, running on
>>>>    Linux, Mac OSX, and Windows.  It used GLEW
>>>>    (http://glew.sourceforge.net/) to dispatch to OpenGL functions, and
>>>>    GLUT (http://www.opengl.org/resources/libraries/glut/) to interact
>>>>    with the window system.  Neither GLEW nor GLUT are compatible with
>>>>    GLES, Wayland, Android, or core profiles.
>>>>
>>>>    Only a few GLES tests existed, but they were compiled separately from
>>>>    the GL tests, and they were not integrated tightly into the framework.
>>>>    Furthermore, tests that in theory should have been able to be run on
>>>>    both GL and GLES were only compiled for GL.
>>>>
>>>>    Long term plan (and status):
>>>>
>>>>    (5) In order for the framework to know which window systems, GL APIs,
>>>>    and extensions are relevant to any particular test, we will add a
>>>>    declarative mechanism for each test to specify the test requirements.
>>>>    The declarative syntax will use macros or inline functions.  For
>>>>    example, if a test should run on desktop GL implementations that
>>>>    support GL_EXT_framebuffer_object, desktop GL implementations version
>>>>    3.0 or later, and on ES2 implementations, the test would contain a
>>>>    line that looks something like "test_params.requirements =
>>>>    OR3(AND(GL_API, EXTENSION("GL_EXT_framebuffer_object")),
>>>>    GL_API_VERSION(3, 0), GLES2_API)".  This will replace the imperative
>>>>    mechanism that we currently use to limit a test to certain GL versions
>>>>    and extensions (the piglit_require_xxx() functions).  Status: not yet
>>>>    started.
>>>>
>>>
>>> This makes it only slightly easier that using piglit_is_supported
>>> functions. That was mostly reason why I added runtime checks for
>>> platform to GLX and EGL code.
>>>
>>> Of course it is nice to write common and possible complex requirements
>>> for tests in standard simple to write fashion. But most likely tests
>>> will still have to do runtime branching depending on if it is running
>>> on GLES or GL.
>>
>> Having the test requirements stated declaratively allows us to do several things
>> that are difficult, or not possible, with the piglit_is_supported functions. Again,
>> these things have been part of Ken's, Paul's, and my plan for some time,
>> but we forgot to document it.
>>
>> 1) Filtering tests by GL API.
>>
>> Eventually, a large portion of tests will be capable of running under GL and GLES.
>> Once we reach that point, developers will normally not want to run all those
>> tests twice, first as GL and then as GLES. The framework will need to be intelligent
>> enough to, by request, run only GL test cases, only GLES test cases, or run GL and GLES test
>> cases (in which case some tests will be ran multiple times).
>>
>> To allow this, the plan is to add some standard command-line option to test
>> executables and to the Piglit python scripts, such as --gl-api. If --gl-api=gles2
>> is given to a test executable, then it will run only its GLES2 test cases. Given
>> --gl-api=all, it will run test cases for all APIs. This plan is not yet set in
>> stone, and we're receptive to suggestions.
>>
>> Similarly, if --gl-api=gles2 is given to piglit-print-commands.py, then the script
>> will  1) invoke each test executable with a flag that requests it to print out info
>> about its requirements, and 2) filter out tests that do no support GLES2. This
>> will give developers fine grained control over which tests they run, control
>> that would be cumbersome to provide with the piglit-run.py's -x option. And this
>> gives QA the ability to filter out inappropriate test cases when running Piglit
>> on Android.
>>
>> 2) Filtering window system platforms.
>>
>> Once support for the --gl-api option is implemented, the next logical step
>> is to add a --platform={all,glx,wayland,x11_egl} option that behaves analogously.
>>
> 
> yes. Standard checks are good way. If I understand your proposal
> correctly tests binary is supposed to tell about if it is run able in
> given platform.
> 
> Wouldn't that makes it simpler to have same tests set in all
> platforms and GL flavors? Tests automatically skip when given
> environment doesn't match what test requires. Fork-exec-skip should be
> fast enough not to affect total runtime much.
> 
>>>>    (6) Since standard C doesn't allow functions to be called inside a
>>>>    global initializer, to facilitate (5) we will replace the main()
>>>>    function in tests/util/piglit-framework.c with a piglit_run() function
>>>>    which takes as its input a data structure describing the test
>>>>    requirements (as well as other initialization info like the initial
>>>>    window size).  Each individual test will contain a main() function
>>>>    that populates the data structure and then calls piglit_run().  To
>>>>    avoid having to write a lot of boilerplate code in each individual
>>>>    test, we'll provide a macro that creates a standard main() function
>>>>    for those tests that don't need a lot of customization.  Status: not
>>>>    yet started.
>>>>
>>>
>>> There is failry simple way to have global initializer functions with
>>> gcc and msvc. But I still fail to see why we would need anything
>>> except piglit-framework.c having main. The framework code can check
>>> for test_params.requirements set by test code. It would work exactly
>>> same as piglit_width etc variables work.
>>
>> You're right. The need for another global initializer function does not
>> neccessitate that each test define main(). The initialization of the
>> requirements struct could just as well go into a function named
>> piglit_require().
>>
>> However, we have to remove main() from libpiglitutil, anyway.
>>
>> The EGL tests, glean, and libpiglitutil define main(). The EGL tests and glean
>> link to libpiglitutil. Surprisingly, the linker doesn't complain. This situation
>> is very fragile. If you modify the libraries or tests in a way that changes
>> the order in which the linker resolves things, link failure occurs. I've encountered
>> such link failures twice already.
>>
> 
> What kind of linking failures?
> 
> It shouldn't fail for main at least because main in binary should
> always override main in a shared library.
> 
> It is well defined which function will be selected. The search is depth
> first search though linked libraries and their dependencies.
> 
> Of course keeping depth first search ordering correct if multiple
> libraries define same symbols and depend on each others can be
> sometimes hard.
> 
>>
>> This situation is absurd. main() needs to be removed from libpiglitutil. This gives
>> us two options: a) that each test define main(), or b) that we create a new static library,
>> libpiglitutil_main, that tests link to.
>>
>> I don't see a strong argument for either (a) or (b), but I think (a) is the simpler approach.
>> Which do you prefer?
>>  
> 
> New static library doesn't solve anything. It won't even allow using
> shared libraries in windows. At least I don't know how symbol overriding
> for piglit_with etc would work in windows.
> 
> But option A is simple to implement. We can have PIGLIT_MAIN macro
> (like boost tests have for windows compatibility) which declares
> default main implementation that calls piglit_run. If tests needs to
> override initialization and/or main loop then it can implement main
> like they do now.

Exactly. We should have a PIGLIT_PLEASE_GIVE_ME_THE_DEFAULT_MAIN(),
and require test authors to write main() only if they need to override
default behavior.

> How about option C where tests would be shared libraries and
> single runner binary would open them with dlopen&dlsym?

This is an interesting idea and might be worth exploring. However,
I think there is enough global change occuring in Piglit at the moment.
We should work towards finishing the current plans before beginning
another large task like this. Let's revisit this idea after tests have the
ability to declaratively state their requirements and the test runner is
able to filter according to those requirements.

- ----
Chad Versace
chad.versace at linux.intel.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJPxqgFAAoJEAIvNt057x8iOlcQAIVBsfV/j00JlYGnQuAOeq1u
5XTrFP+igt5QR81eWSGwJuqtYZB6Ih8Nqfl1KyeGTzEHOfvzRGaoefF89LAsuwag
AVL97pEo5A9z/MaPOl2Plj3CNX8OjLU9BJiPvX++SjMfS9WZ9xgUlrJer5Wrl9Ly
QwY35dB/huD2o6ueM1Ip4/ifbN+fXbPLDkhC9YytCodATSHNNtNUnRpNvLn1xz+g
Jl2DOQm6O3ViSVv2GlwsNyV7iiObfRVgSd+iPEwysdjSCGU19yz+JoX6d6ZnF0hn
ZtVL2mWI466xBLzACiRn0h/0H1sqzxgEHV1UpUY77bBCVa3wMNOQr5e5yrAzeZiL
y/Kxx24pybFpShhDT068xsQK4MkqrOpEaeC+nLkjriVvfJYmZhCGt5MFfRdbcWQj
umB4NQED//sCEtUoVhL6tXWUlA7Kf9ypiOPwOystbMANmcOCrpSkSbiwb24/sOZb
oNwBazOcvS/jyWfNMINx2AePn6hCoZ6pTsJZHEIrXnU0noDrTTYVjfW2KzgrDph8
wjldCFyOa3BdWvtOEAELuFPiXr4BT40GQN8CLAWH+6816MUxI6DWLAXVJuOFIEON
2mMPzpoaRU/K29xSxR8b6jSobp3KBTzqHr2wAznpGwMTQN/DCzZklp85qf3nV9Qa
YSxPq+8TxxtkqPQ3CqmQ
=nzOQ
-----END PGP SIGNATURE-----