[Piglit] Add platform based test skipping to python
baker.dylan.c at gmail.com
Fri Oct 3 17:57:41 PDT 2014
We have recently started using jenkins here at Intel to do regression
testing across multiple generations of hardware. Because of this we have
become concerned with the large number of tests that fail and the large
number of tests that skip, specifically on older platforms.
This series is the first 15 patches shaved off of a larger series I'm
working on to add better test skipping in the framework.
I have targeted the framework as the best place to skip tests for a
couple of reasons. First, it allows us to use metadata in the various
profiles as skip criteria; in this series platforms (glx, egl, gbm, etc)
are used to skip tests when that platform isn't available. In a later
patches I'm working on we're able to use GL support as that criteria,
allowing old platforms to skip GL features that were not part of that
hardware's targeted GL API.
Doing this work I took some time to cleanup the piglit Test code, I
created a test package to hold the growing number of Test classes, split
some code into Mixins, extended the early return code in Test.run() to
be more versatile, split the PiglitTest class into PiglitGLTest and
PiglitCLTest classes, pulled the code that adds -fbo and -auto into
PiglitGLTest's command getter, and cleaned up some potential bugs.
All of these changes result in better code, better code sharing between
OpenGL and OpenCL tests (which means basically OpenCL doesn't run OpenGL
specific code which only increases their runtime), and ultimately the
ability to skip tests based on platform.
This series implements fast skipping for two types of tests, glx tests
on non glx, and egl tests on glx. I was somewhat unsure about the right
way to skip the egl tests. Currently all platforms except pure glx
support egl (x11_egl, mixed_glx_egl, and wayland always do), or can be
configured with egl support (gbm). I chose to exclude glx, but I would
be equally opened to include x11_egl, mixed_glx_egl, gbm, and wayland
The difference in runtime for a test that fast skips vs one that slow
skips is dramatic. I measured that it takes about .00005 seconds for a
test to fast skip, and about .1 seconds to slow skip. That adds up very
quickly if your platform skips 20000 tests.
More information about the Piglit