[Piglit] piglit-dispatch for gles

[Tom Gall]

On Wed, Jan 2, 2013 at 7:16 PM, Paul Berry <stereotype441 at gmail.com> wrote:
> On 2 January 2013 15:28, Tom Gall <tom.gall at linaro.org> wrote:
>>
>> Hi All,
>>
>> I was looking at some of the piglit-dispatch code experimenting a bit
>> with the intent to implement it for gles and was wondering where did
>> the past discussions on this end up?
>>
>> There is a comment above piglit_dispatch_default_init that mentions
>> run-time parameters. The "Waffle and piglit-dispatch plans" thread
>> from back in May + June 2012 are of interest but I'm not sure
>> consensus was reached.
>>
>> Was it? Is there any sort of design put together beyond what's
>> currently in the code?
>
>
> Your timing couldn't be better, Tom.  I just started to work on this again
> this week.  I would love to hear what you've discovered in your experiments,
> and I'm happy to split the implementation effort with you if you're
> interested.

Great! I'm glad to hear you are putting time into this and I'd
certainly like to lend a hand as would make sense.

> Here's what I would like to have in the long term:
>
> 1. Waffle needs a facility like piglit-dispatch, so rather than duplicate
> code between piglit and waffle, it makes sense to migrate piglit-dispatch to
> waffle, and then extend it to support gles.

Exactly!

> 2. Piglit-dispatch, in its current incarnation, generates its code based on
> the files gl.spec, gl.tm, enumext.spec (all published at
> www.opengl.org/registry/api/).  However, there are some errors and omissions
> in those files, so to work around those errors, Piglit has its own copy of
> them, which we have to maintain independently.  This has proven to be kind
> of a mess to keep updated when opengl.org publishes new files, and it's only
> going to get worse when we try to add information about gles.  So I'm
> currently working on some scripts that attempt to pull information from
> multiple sources (the aforementioned files, the standard gles headers,
> mesa's XML files, and perhaps other sources), and produce a master database
> of all GL/GLES functions that we can base waffle-dispatch on.

This sounds potentially problematic but I agree having this all be
auto generated is the way to go rough edges and all.

> I'm currently in the middle of item 2, and I think I will have something
> worth showing to the public within about a week.  Add another week to port
> piglit-dispatch to waffle and get it building, another week to compensate
> for developer bravado, and another week or two for things to get reviewed on
> various mailing lists, and my rough guess is that things will probably start
> working sometime in February.

Good plan, might be a little ambitious schedule taking in account
ancillary bits but who cares, it'll be done when it's done.

I like the parse_glspec.py and json etc approach. That seems spot on.

I'm still pulling apart how what is currently there works but all in
all seems straight forward and I'm not finding any pixie dust or
unicorn tears.

My incomplete list, trying to break the problem apart, I had:
0) add param to piglit-run.py to define which api / version
combinations should be run
  -)  in python there needs to be a way to to define a test as valid
for combinations of api / version. Just trying everything blindly
seems wrong.
     Maybe there is a new sort of 'invalid' piglit result, causing the
test to not even be listed in the results.
  -) The summary side of things will need to handle multiple api/version runs.

1) need a mechanism to discover api (gl/gles) & versions available at runtime.
   -) Within tests/util a mechanism to set what api/version piglit and
test will run as.  Probably becomes part of each test's main().

your #2.

3)  (optional) Dispatch could allow one to pass a function pointer
that will be called in the case of a failure. Ex: dispatch tries to
call a fn which for say gles does exist, call user supplied fn and in
that fn they can decide to immediately return PIGLIT_FAIL, try to
continue on or whatever.

Obviously the holy grail here seems to be will it be possible to have
a universal mechanism with gles and gl in one dingus that also takes
into account versions such that when testcases run they can be built
but once and yet at runtime specify an api/version.

Testcases like shader_runner, for instance I think will handle being
"universal" between gl / gles quite well.

But, aren't there are perfectly valid class of testcases which are
targeting specific a api/version and probably shouldn't be built/run
any other way? To me this would seem to suggest that there might very
well be two classes of tests. Universal and api specific that might
not use the dispatcher.

If you hadn't already begun, I was thinking a more humble approach
starting out with a very limited set of the APIs and a very limited
set of testcases which were good universal candidates and iterate,
iterate, iterate. (minmax, shader_runnerer seemed like good candidates
to me)

> I was planning to discuss these plans with some of the other Intel folks
> when I'm back in the office tomorrow.  Feel free to chime in with your
> thoughts.

Sounds good. Exciting stuff.

--
Regards,
Tom

"Where's the kaboom!? There was supposed to be an earth-shattering
kaboom!" Marvin Martian
Graphics Working Group | Linaro.org │ Open source software for ARM SoCs
w) tom.gall att linaro.org
h) tom_gall att mac.com