[Piglit] Replacing GLUT -- Testing ES, EGL, core profiles, Wayland, etc

[Chad Versace]

On 04/10/2012 03:34 AM, Jose Fonseca wrote:
> 
> 
> ----- Original Message -----
>> I propose that we replace GLUT with something that meets Piglit's
>> needs.
>> I want to hear your opinion on this.
>>
>> Problems with Piglit
>> --------------------
>>
>> Piglit has several inadequacies that need to be addressed. Some
>> inadequacies,
>> such as poor support for ES tests, have been an inconvenience that
>> the Mesa
>> devs have succeeded in overlooking so far (and it's embarrassing how
>> long
>> we've been content to overlook it). Others, such as no support for
>> Wayland, or
>> core profiles, have not become a problem yet, but will become pain
>> points in
>> the near future if not addressed.
>>
>> Here are the major inadequacies I see that need to be fixed soon:
>>     - Piglit does not support core profiles.
>>
>>       This will prevent us from testing core 3.x profiles in Mesa
>>       when we
>>       eventually begin implementing them.  And today, this prevents
>>       us from
>>       running GL 3 tests on Apple.
>>
>>     - Piglit doesn't really support ES tests.
>>     
>>       We have a few tests, specific to GLES1 and GLES2, scattered
>>       about, but
>>       we have no coherent way to write ES tests and integrate them
>>       into the
>>       framework.
>>
>>       Many of Piglit's GL tests could just as well run on GLES2 with
>>       a little
>>       tweaking. But to take advantage of that, Piglit needs a method
>>       to run
>>       a single test under different API's. (Paul's work on
>>       piglit-dispatch was
>>       a prerequisite for this).
>>
>>     - Piglit does not support X11/EGL.
>>
>>       Mesa has an implementation of libEGL, but we don't really test
>>       it. There
>>       is not way to run the GL tests with X11/EGL; they run only on
>>       GLX.
>>       Piglit does have a few scattered tests for EGL extentions, but
>>       that's
>>       not the same as supporting EGL as a first-class platform.
>>
>>     - Piglit does not support Wayland or Android.
>>
>>
>> What a solution might look like
>> -------------------------------
>>
>> I think the first step to fix these problems is to replace GLUT with
>> something
>> more flexible. I mean *really* flexible, something that will allow
>> individual
>> tests to select a GL flavor (api/version/profile) and a window system
>> (GLX,
>> X11/EGL, Wayland, etc) at runtime.  Now that piglit-dispatch has
>> eliminated
>> the need for tests to link to libGL and include GL headers, there is
>> a real
>> possibility that we can make this happen.
>>
>> This will eventually allow us to write *one* test, annotate which GL
>> flavors
>> it is compatible with in a comment block (like the glslparser tests),
>> and run
>> it with different API's and winsys. For example, if a test was
>> properly
>> annotated, libpiglitutil would allow running it once like this
>>   ./test --gl-api=gl --gl-profile=core --gl-version=3.2 --winsys=glx
>> and again like this
>>   ./test --gl-api=gles2 --winsys=x11_egl
>> (Those options are from a brainstorm session Ken and I had. There's
>> no working
>> code that does that).
> 
> 
> 
>  
>> I originally considered that Piglit should keep GLUT, and that I
>> should just
>> contribute the necessary features to GLUT upstream. But I feel that
>> a better choice is to abandon GLUT altogether because
>>     1) Piglit's needs, such as dynamic choice of GL flavor and
>>     winsys, are
>>        outside of GLUT's goals. And those needs may need to evolve at
>>        a pace
>>        that upstream GLUT is uncomfortable with.
>>     2) GLUT has a lot of feature cruft that Piglit doesn't care
>>     about. I don't
>>        want to implement and maintain that cruft for Wayland and
>>        possibly
>>        other future platforms.
>>
>>
>> Waffle as a replacement for GLUT
>> --------------------------------
>>
>> My desired goal is that we replace GLUT with a lightweight library
>> tailored to
>> meet Piglit's needs, not with another GLUT clone.  I started a new
>> project,
>> Waffle, to be that replacment.
>>
>> homepage:
>>       http://people.freedesktop.org/~chadversary/waffle/index.txt
>> source:         git://people.freedesktop.org/~chadversary/waffle.git
>> gitweb:         http://cgit.freedesktop.org/~chadversary/waffle
>>
>> I want to hear people's concerns and opinions about 1) replacing GLUT
>> with
>> a library that enables the "write one test, run many API's" goal
>> discussed
>> above, about 2) how suitable Waffle is as a replacment, and about 3)
>> bad
>> design choices I may have made in the Waffle API. If you like the
>> idea of
>> Waffle, please voice your opinions on it. I want its API and
>> feature-set to
>> meet everyone's Piglit needs, not just my short-sighted needs.
> 
> Chad,
> 
> FWIW, this all looks quite sensible to me.
> 
> I agree that piglit requirements are disjoint enough to justify something other than glut.
> 
> Your design covers my main concern of portability (both the design and build system chosen should allow to easily support Windows).

Thanks for taking the time to examine it.

> For waffle to make sense as a standalone project and more useful for GL tools other than piglit then I think that the GL call dispatch mechanism should also be folded inside waffle too (maybe an an optional thing).

I agree. For waffle to be a viable project independent of Piglit, Piglit's dispatch mechanism needs to be either folded into waffle directly or be made into an independent project on which waffle depends.

When Paul was first designing piglit-dispatch, he and I discussed this. Paul first wanted to get piglit-dispatch working smoothly inside Piglit before separating it as an independent project. If I recall correctly, I believe he also wanted to get piglit-dispatch working well with GLES before the separation. Maybe he can comment on his plans here.

> I'd definitely would like to use waffle for apitrace's test suite, and maybe eventually replace apitrace's glws abstraction in glretrace too.

It's good to hear that waffle has potential clients other than piglit! One of the things I hope to use waffle for in the future is to take an api trace of a game on Mac (on which there are many more interesting workloads available than on Linux), and use waffle to replay it on Linux as a workload. I haven't yet looked into apitrace's glws abstraction layer. I should do that soon.
 
> One thing to keep in mind while designing this is cross-API interoperability. Once we take care of getting piglit tests running either on GL/GLES1/GLES2, we will want to write tests that test interoperability between GL and GLES1 and GLES2. And waffle should allow that.

Thanks for bringing up the issue of cross-API interoperability. That is not something I considered, and thought no one was interested in it. Accordingly, waffle's design doesn't support that now. After I complete waffle's GLX backend, I'll revisit this and redesign the API to allow it. Expect to receive RFC's from me.

----
Chad Versace
chad.versace at linux.intel.com