[Mesa-dev] deqp-runner "suite" support

Emma Anholt emma at anholt.net
Thu Jul 22 19:00:31 UTC 2021

One of the irritating parts of deqp testing in our CI is that
deqp-runner has only been able to invoke one deqp binary per
deqp-runner invocation, which meant that we have to split out each of
deqp-gles*, deqp-egl, and glcts into different CI jobs, producing a
whole lot of output for devs to review when things go wrong (or for me
to review when trying to find slow testcases to optimize).  It also
wastes hardware time bringing up the test environment for each CI job,
and makes job tuning to meet our ~10-minute runtime goal a lot of

So, I'm adding support to deqp-runner to describe a test suite in a
toml file and have it run all the deqp-*s in parallel.  I'm already
enjoying it for doing full, quick softpipe CTS runs locally.  I'm
hoping to merge it next week when I get back from camping, so I
thought I'd put the MR up here for feedback on the interface:

deqp-runner code:

Mesa commit showing its use:

Test Mesa pipeline:

Should any of the toml keys be renamed?  Should anything else be in
command line args? Do we need toml top-level settings for "apply this
to all the [[deqp]]s"?  Should I bite the bullet and let you do both
deqp and piglit in the same suite config (new runner binary?)

More information about the mesa-dev mailing list