[Piglit] Add platform based test skipping to python

[Dylan Baker]

On Wednesday, October 15, 2014 10:11:18 AM Emil Velikov wrote:
> Hi Dylan,
> 
> On 04/10/14 01:57, Dylan Baker wrote:
> > We have recently started using jenkins here at Intel to do regression
> > testing across multiple generations of hardware. Because of this we have
> > become concerned with the large number of tests that fail and the large
> > number of tests that skip, specifically on older platforms.
> > 
> > This series is the first 15 patches shaved off of a larger series I'm
> > working on to add better test skipping in the framework.
> > 
> > I have targeted the framework as the best place to skip tests for a
> > couple of reasons. First, it allows us to use metadata in the various
> > profiles as skip criteria; in this series platforms (glx, egl, gbm, etc)
> > are used to skip tests when that platform isn't available. In a later
> > patches I'm working on we're able to use GL support as that criteria,
> > allowing old platforms to skip GL features that were not part of that
> > hardware's targeted GL API.
> > 
> > Doing this work I took some time to cleanup the piglit Test code, I
> > created a test package to hold the growing number of Test classes, split
> > some code into Mixins, extended the early return code in Test.run() to
> > be more versatile, split the PiglitTest class into PiglitGLTest and
> > PiglitCLTest classes, pulled the code that adds -fbo and -auto into
> > PiglitGLTest's command getter, and cleaned up some potential bugs.
> > 
> > All of these changes result in better code, better code sharing between
> > OpenGL and OpenCL tests (which means basically OpenCL doesn't run OpenGL
> > specific code which only increases their runtime), and ultimately the
> > ability to skip tests based on platform.
> > 
> > This series implements fast skipping for two types of tests, glx tests
> > on non glx, and egl tests on glx. I was somewhat unsure about the right
> > way to skip the egl tests. Currently all platforms except pure glx
> > support egl (x11_egl, mixed_glx_egl, and wayland always do), or can be
> > configured with egl support (gbm). I chose to exclude glx, but I would
> > be equally opened to include x11_egl, mixed_glx_egl, gbm, and wayland
> > instead.
> > 
> I would prefer if we go with the explicit "requires_platform", otherwise
> as we add wgl and cgl we'll end up in a mess :\ Perhaps adding a meta
> platform "egl" to encapsulate x11_egl, mixed_glx_egl, gbm, and wayland
> might be a good idea ?
> 
> Thanks
> Emil

Well, you could easily add cgl and wgl to that list since both
requires_platforms and exclude_platforms both are a list argument. I do
however agree that having some meta options might be useful as well.

> 
> > The difference in runtime for a test that fast skips vs one that slow
> > skips is dramatic. I measured that it takes about .00005 seconds for a
> > test to fast skip, and about .1 seconds to slow skip. That adds up very
> > quickly if your platform skips 20000 tests.
> > 
> > _______________________________________________
> > Piglit mailing list
> > Piglit at lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/piglit
> > 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.freedesktop.org/archives/piglit/attachments/20141015/d61d377a/attachment.sig>