[Piglit] Is it normal to see piglit quick.tests results changing?

[Jose Fonseca]

I'm not sure if this is the case, but often driver bugs cause undefined/garbagge rendering, which may some times pass by mere chance. Other times, rendering is not the problem, but race conditions between driver/hw.

Random test results are very bad -- as they are distracting and misleading -- so I think they should be either excluded from the periodic tests , or (even better) have the driver fixed so they pass consistently.

Jose

----- Original Message -----
> Hi,
> 
> I noticed that if I run piglit quick.tests twice, some tests would
> change from pass to fail and vice versa:
> ex:
>    piglit-run /usr/lib64/piglit/tests/quick.tests result0
>    piglit-run /usr/lib64/piglit/tests/quick.tests result1
>    piglit-summary-html summary results0 results1
>    xdg-open summary/index.html
> 
> I see these tests changing:
>    fbo-3d
>    fbo-sys-blit
>    fbo-sys-blit
>    bgra-sec-color-pointer
>    scissor-clear
>    (... quite a bit more ...)
> Is this expected?
> 
> I uploaded the piglit-summary-html report to
> http://scottt.tw/piglit-summary/index.html
> This is on an Intel Sandy Bridge machine running Fedora 16
> (kernel 3.1, mesa-7.11 from distro packages):
> http://www.smolts.org/client/show_all/pub_19218c58-fdb9-4c7e-81d7-c0d1e5b8c17d
> 
> Matěj Cepl plans to use piglit for internal QA at Red Hat and I'm
> helping him getting it packaged for Fedora. Could anyone share some
> experience on using piglit for regression testing?
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/piglit
>