[Piglit] Is it normal to see piglit quick.tests results changing?

[Ian Romanick]

On 01/14/2012 01:01 PM, Scott Tsai wrote:
> Hi,
>
> I noticed that if I run piglit quick.tests twice, some tests would
> change from pass to fail and vice versa:
> ex:
>     piglit-run /usr/lib64/piglit/tests/quick.tests result0
>     piglit-run /usr/lib64/piglit/tests/quick.tests result1
>     piglit-summary-html summary results0 results1
>     xdg-open summary/index.html
>
> I see these tests changing:
>     fbo-3d
>     fbo-sys-blit
>     fbo-sys-blit
>     bgra-sec-color-pointer
>     scissor-clear
>     (... quite a bit more ...)
> Is this expected?

No, absolutely not.  I've seen some of the GLX tests and time-elapsed 
intermittently change state, but all of the others are rock-solid for 
me.  Our QA group runs piglit on Mesa every night for regression 
testing, and I they see the same results I see.

Out of curiosity, do you see stable results if you run on a bare xserver 
(i.e., no compositing window manager)?

> I uploaded the piglit-summary-html report to
> http://scottt.tw/piglit-summary/index.html
> This is on an Intel Sandy Bridge machine running Fedora 16
> (kernel 3.1, mesa-7.11 from distro packages):
> http://www.smolts.org/client/show_all/pub_19218c58-fdb9-4c7e-81d7-c0d1e5b8c17d
>
> Matěj Cepl plans to use piglit for internal QA at Red Hat and I'm
> helping him getting it packaged for Fedora. Could anyone share some
> experience on using piglit for regression testing?