[Piglit] piglit-dispatch for gles

[Chad Versace]

On 01/03/2013 10:00 AM, Tom Gall wrote:

> I'm still pulling apart how what is currently there works but all in
> all seems straight forward and I'm not finding any pixie dust or
> unicorn tears.
> 
> My incomplete list, trying to break the problem apart, I had:
> 0) add param to piglit-run.py to define which api / version
> combinations should be run
>   -)  in python there needs to be a way to to define a test as valid
> for combinations of api / version. Just trying everything blindly
> seems wrong.

Yes, we definitely need something like that. But I think the metadata
should go into the testfile itself.

Piglit's history has proved it unwise to maintain much test metadata outside
of the testfile and in the python. The python scripts use to contain much more
metadata. For example, the GLSL version and expected result for glslparser tests
were kept in all.tests (see commits 6c7d874b and 71bb5333). The problem was
that the metadata sometimes drifted out-of-sync with the actual test.

One requirement for encoding the metadata in the testfile is that there must
be a method for the python scripts to easily extract that metadata.

I don't have a firm plan yet on how to accomplish that, but I do have tentative, messy
plan. As the final step of the build phase, the python scripts would run each executable
with a special flag, --give-me-json-that-tells-me-what-gl-flavors-you-support, and
the scripts would cache the result. Then, when you run piglit-run.py, it would refer to
the cached json in order to determine which set of gl flavors over which to iterate the
test.

That plan isn't perfect, but after much discussion with Ken Graunke and Paul,
we were unable to formulate a better one. Any suggestions?

>      Maybe there is a new sort of 'invalid' piglit result, causing the
> test to not even be listed in the results.
>   -) The summary side of things will need to handle multiple api/version runs.

Yes. Wow... we even need to do some html hacking to achieve our goal.
When you look at all the necessary steps at once, as you laid it out,
it becomes apparent that the route to achieve the ultimate goal involves a
lot of detours and is not as it simple  as it first seems.

> 1) need a mechanism to discover api (gl/gles) & versions available at runtime.

Waffle has a function for detecting that: waffle_display_supports_context_api()
[http://people.freedesktop.org/~chadversary/waffle/man/waffle_display.3.html] .

>    -) Within tests/util a mechanism to set what api/version piglit and
> test will run as.  Probably becomes part of each test's main().

Most of the infrastructure for that mechanism is already present in piglit-framework.
The deficiencies in the current infrastructure is that:
  - The API is chosen at build-time.
  - Only one version may be assigned to each API.


> 3)  (optional) Dispatch could allow one to pass a function pointer
> that will be called in the case of a failure. Ex: dispatch tries to
> call a fn which for say gles does exist, call user supplied fn and in
> that fn they can decide to immediately return PIGLIT_FAIL, try to
> continue on or whatever.

Yes, (3) is in the plan. waffle-dispatch will be carefully tailored for
piglit's needs, but be designed to still be useful to other potential
clients. Therefore a hook is definitely needed for dispatch failure.

> 
> Obviously the holy grail here seems to be will it be possible to have
> a universal mechanism with gles and gl in one dingus that also takes
> into account versions such that when testcases run they can be built
> but once and yet at runtime specify an api/version.
> 
> Testcases like shader_runner, for instance I think will handle being
> "universal" between gl / gles quite well.
> 
> But, aren't there are perfectly valid class of testcases which are
> targeting specific a api/version and probably shouldn't be built/run
> any other way? To me this would seem to suggest that there might very
> well be two classes of tests. Universal and api specific that might
> not use the dispatcher.

Even though there may exist many GL/GLES tests that could fall into the
api-specific category, for simplicity we should probably place all GL/GLES
into the build-once-and-dispatch-dynamically catgeory. I think that the only
tests that should really go into the api-specific category are those that
test windowing api's, such as the GLX and EGL tests.

> Exciting stuff.

Totally.