[Piglit] Newbie question: command to run tests that passed before?

[Mark Janes]

Dan Kegel <dank at kegel.com> writes:

> Hrm.  A solution that involved Jenkins would be way overkill.
> I'm just looking for the little core idiom for running just the
> test of tests that passed in the, um, past.

Even if you don't want to set up a CI with jenkins, you can refer to the
automation that I've written at https://github.com/janesma/mesa_jenkins

Our approach is to list all failing/crashing tests in a config file, and
pass that to piglit.  Piglit converts "known failing" tests to a "skip"
status.  The only tests we can't run are the ones that have flaky
results or cause gpu hang.

For example:
https://github.com/janesma/mesa_jenkins/blob/master/piglit-test/bdw.conf

> Maybe that's not the way people do it?  Does everyone just compare
> logs against old logs?
>
> Thanks, and say hi to Ethyl for me.
> - Dan
>
> On Wed, Sep 7, 2016 at 4:08 PM, Ilia Mirkin <imirkin at alum.mit.edu> wrote:
>> [+janesma]
>>
>> Mark has set up a CI system at Intel, and I'm fairly sure did a
>> writeup about it and shared code, but I can't for the life of me
>> remember the details. Maybe a few Guinesses ago, but definitely not
>> now... hopefully he can share. [It involves Jenkins.]
>>
>>   -ilia
>>
>> On Wed, Sep 7, 2016 at 6:59 PM, Dan Kegel <dank at kegel.com> wrote:
>>> Hi!
>>> I'd like to use piglit as a regression test for my opengl test rigs.
>>> My plan is to run piglit a few times on each OS/card combo I need to support,
>>> get a list of tests that reliably pass on each platform, and then
>>> forevermore run just those tests.
>>>
>>> I know about the -t and -x options, and the --test-list option, and
>>> about writing test profiles,
>>> but am still not sure what the best way to generate a list of passing
>>> tests to pass to e.g. --test-list.
>>> Test names are a little hard for this newbie to identify in the output
>>> of piglit summary console.
>>>
>>> After looking at the source a bit, I tried generating a list of passed
>>> tests like this:
>>>     ./piglit summary console results/quick | grep ': pass$' | sed
>>> 's/:.*//' | tr / @ > tests.list
>>> but that includes output
>>> spec@!opengl 1.1 at max-texture-size-level
>>> spec@!opengl 1.1 at max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16
>>>
>>> which leads to errors from --test-list tests.list like
>>> Fatal Error: Cannot reorder test: "spec@!opengl
>>> 1.1 at max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16", it is not in the
>>> profile.
>>>
>>> Suggestions?
>>>
>>> p.s. Mesa Haswell is behaving very well on stock ubuntu 16.04, only 1%
>>> of quick tests fail, and there are no hangs or kernel complaints.
>>> _______________________________________________
>>> Piglit mailing list
>>> Piglit at lists.freedesktop.org
>>> https://lists.freedesktop.org/mailman/listinfo/piglit