[Piglit] Replacing GLUT -- Testing ES, EGL, core profiles, Wayland, etc

[Jose Fonseca]

----- Original Message -----
> On 04/10/2012 03:34 AM, Jose Fonseca wrote:
> > 
> > 
> > ----- Original Message -----
> >> I propose that we replace GLUT with something that meets Piglit's
> >> needs.
> >> I want to hear your opinion on this.
> >>
> >> Problems with Piglit
> >> --------------------
> >>
> >> Piglit has several inadequacies that need to be addressed. Some
> >> inadequacies,
> >> such as poor support for ES tests, have been an inconvenience that
> >> the Mesa
> >> devs have succeeded in overlooking so far (and it's embarrassing
> >> how
> >> long
> >> we've been content to overlook it). Others, such as no support for
> >> Wayland, or
> >> core profiles, have not become a problem yet, but will become pain
> >> points in
> >> the near future if not addressed.
> >>
> >> Here are the major inadequacies I see that need to be fixed soon:
> >>     - Piglit does not support core profiles.
> >>
> >>       This will prevent us from testing core 3.x profiles in Mesa
> >>       when we
> >>       eventually begin implementing them.  And today, this
> >>       prevents
> >>       us from
> >>       running GL 3 tests on Apple.
> >>
> >>     - Piglit doesn't really support ES tests.
> >>     
> >>       We have a few tests, specific to GLES1 and GLES2, scattered
> >>       about, but
> >>       we have no coherent way to write ES tests and integrate them
> >>       into the
> >>       framework.
> >>
> >>       Many of Piglit's GL tests could just as well run on GLES2
> >>       with
> >>       a little
> >>       tweaking. But to take advantage of that, Piglit needs a
> >>       method
> >>       to run
> >>       a single test under different API's. (Paul's work on
> >>       piglit-dispatch was
> >>       a prerequisite for this).
> >>
> >>     - Piglit does not support X11/EGL.
> >>
> >>       Mesa has an implementation of libEGL, but we don't really
> >>       test
> >>       it. There
> >>       is not way to run the GL tests with X11/EGL; they run only
> >>       on
> >>       GLX.
> >>       Piglit does have a few scattered tests for EGL extentions,
> >>       but
> >>       that's
> >>       not the same as supporting EGL as a first-class platform.
> >>
> >>     - Piglit does not support Wayland or Android.
> >>
> >>
> >> What a solution might look like
> >> -------------------------------
> >>
> >> I think the first step to fix these problems is to replace GLUT
> >> with
> >> something
> >> more flexible. I mean *really* flexible, something that will allow
> >> individual
> >> tests to select a GL flavor (api/version/profile) and a window
> >> system
> >> (GLX,
> >> X11/EGL, Wayland, etc) at runtime.  Now that piglit-dispatch has
> >> eliminated
> >> the need for tests to link to libGL and include GL headers, there
> >> is
> >> a real
> >> possibility that we can make this happen.
> >>
> >> This will eventually allow us to write *one* test, annotate which
> >> GL
> >> flavors
> >> it is compatible with in a comment block (like the glslparser
> >> tests),
> >> and run
> >> it with different API's and winsys. For example, if a test was
> >> properly
> >> annotated, libpiglitutil would allow running it once like this
> >>   ./test --gl-api=gl --gl-profile=core --gl-version=3.2
> >>   --winsys=glx
> >> and again like this
> >>   ./test --gl-api=gles2 --winsys=x11_egl
> >> (Those options are from a brainstorm session Ken and I had.
> >> There's
> >> no working
> >> code that does that).
> > 
> > 
> > 
> >  
> >> I originally considered that Piglit should keep GLUT, and that I
> >> should just
> >> contribute the necessary features to GLUT upstream. But I feel
> >> that
> >> a better choice is to abandon GLUT altogether because
> >>     1) Piglit's needs, such as dynamic choice of GL flavor and
> >>     winsys, are
> >>        outside of GLUT's goals. And those needs may need to evolve
> >>        at
> >>        a pace
> >>        that upstream GLUT is uncomfortable with.
> >>     2) GLUT has a lot of feature cruft that Piglit doesn't care
> >>     about. I don't
> >>        want to implement and maintain that cruft for Wayland and
> >>        possibly
> >>        other future platforms.
> >>
> >>
> >> Waffle as a replacement for GLUT
> >> --------------------------------
> >>
> >> My desired goal is that we replace GLUT with a lightweight library
> >> tailored to
> >> meet Piglit's needs, not with another GLUT clone.  I started a new
> >> project,
> >> Waffle, to be that replacment.
> >>
> >> homepage:
> >>       http://people.freedesktop.org/~chadversary/waffle/index.txt
> >> source:
> >>         git://people.freedesktop.org/~chadversary/waffle.git
> >> gitweb:         http://cgit.freedesktop.org/~chadversary/waffle
> >>
> >> I want to hear people's concerns and opinions about 1) replacing
> >> GLUT
> >> with
> >> a library that enables the "write one test, run many API's" goal
> >> discussed
> >> above, about 2) how suitable Waffle is as a replacment, and about
> >> 3)
> >> bad
> >> design choices I may have made in the Waffle API. If you like the
> >> idea of
> >> Waffle, please voice your opinions on it. I want its API and
> >> feature-set to
> >> meet everyone's Piglit needs, not just my short-sighted needs.
> > 
> > Chad,
> > 
> > FWIW, this all looks quite sensible to me.
> > 
> > I agree that piglit requirements are disjoint enough to justify
> > something other than glut.
> > 
> > Your design covers my main concern of portability (both the design
> > and build system chosen should allow to easily support Windows).
> 
> Thanks for taking the time to examine it.
> 
> > For waffle to make sense as a standalone project and more useful
> > for GL tools other than piglit then I think that the GL call
> > dispatch mechanism should also be folded inside waffle too (maybe
> > an an optional thing).
> 
> I agree. For waffle to be a viable project independent of Piglit,
> Piglit's dispatch mechanism needs to be either folded into waffle
> directly or be made into an independent project on which waffle
> depends.
> 
> When Paul was first designing piglit-dispatch, he and I discussed
> this. Paul first wanted to get piglit-dispatch working smoothly
> inside Piglit before separating it as an independent project. If I
> recall correctly, I believe he also wanted to get piglit-dispatch
> working well with GLES before the separation. Maybe he can comment
> on his plans here.
> 
> > I'd definitely would like to use waffle for apitrace's test suite,
> > and maybe eventually replace apitrace's glws abstraction in
> > glretrace too.
> 
> It's good to hear that waffle has potential clients other than
> piglit! One of the things I hope to use waffle for in the future is
> to take an api trace of a game on Mac (on which there are many more
> interesting workloads available than on Linux), and use waffle to
> replay it on Linux as a workload.

This should be already possible today with apitrace, minus the fact that Apple tends to support different extensions (GL_foo_APPLE instead of GL_foo_ARB / GL_foo_EXT).
It's definitely possible with Windows apps, on similarly capable GL implementations.

This works because apitrace only replays glFoo verbatim, but always emulates glXFoo, wglFoo, CGLFoo, eglFoo, etc.

Jose