[Piglit] [RFC 00/10] Running dEQP tests without process isolation

[Nicolai Hähnle]

Hi all,

This series allows running dEQP tests (including GL CTS) without process
isolation. I've been using it quite successfully for the last few weeks,
though there are a bunch of loose ends, off the top of my head:

1. One remaining spurious failure in the `gpu` profile
2. Only tested on gpu, deqp_*, cts_gl45, khr_gl45 profiles
3. Unit tests are completely and utterly broken
4. junit backend is untested and likely needs fixes

The reason for all this is that in order to convince Piglit to run dEQP
tests without process isolation, I ended up refactoring a lot of its
internals. Most of this is documented in the commit messages, but a
rough overview:

1. Multiple TestResults can co-exist at the same "root" in the test
   hierarchy. So you can have many TestResults that all have the root
   'deqp-gles31 at functional', and subtests spread out across possibly
   overlapping subgroups. This obviously involves a change in the
   on-disk format for storing results.

2. Test cases are no longer grouped in the profile description. Instead,
   there's the concept of a TestWorkItem, which groups a bunch of tests
   that are to be run in one batch by the same TestRunner.

   This means that the test case objects are no longer responsible for
   running themselves; that's the domain of the separate TestRunner
   hierarchy. TestRunners for shader tests and dEQP tests know how to
   run multiple tests in one batch, and how to parse the test process
   outputs into subtest results.

   Theoretically that should also allow running shader tests from different
   directories together, but I haven't done that in this series.

3. TestWorkItems are grouped deterministically in the profile run logic,
   which means that resumed test runs can still have process isolation
   disabled!

You can find the whole series in context with some other stuff at
https://cgit.freedesktop.org/~nh/piglit/log/?h=deqp-run-multi

I probably won't be able to really polish and clean this up any time
soon, so it would be awesome if somebody would volunteer for this admittedly
unthankful task. In the meantime, the code is out there, and I hope other
people will find it useful.

Cheers,
Nicolai