[Piglit] Is it normal to see piglit quick.tests results changing?

[Eric Anholt]

On Thu, 19 Jan 2012 08:54:34 -0800, Ian Romanick <idr at freedesktop.org> wrote:
> On 01/14/2012 01:01 PM, Scott Tsai wrote:
> > Hi,
> >
> > I noticed that if I run piglit quick.tests twice, some tests would
> > change from pass to fail and vice versa:
> > ex:
> >     piglit-run /usr/lib64/piglit/tests/quick.tests result0
> >     piglit-run /usr/lib64/piglit/tests/quick.tests result1
> >     piglit-summary-html summary results0 results1
> >     xdg-open summary/index.html
> >
> > I see these tests changing:
> >     fbo-3d
> >     fbo-sys-blit
> >     fbo-sys-blit
> >     bgra-sec-color-pointer
> >     scissor-clear
> >     (... quite a bit more ...)
> > Is this expected?
> 
> No, absolutely not.  I've seen some of the GLX tests and time-elapsed 
> intermittently change state, but all of the others are rock-solid for 
> me.  Our QA group runs piglit on Mesa every night for regression 
> testing, and I they see the same results I see.
> 
> Out of curiosity, do you see stable results if you run on a bare xserver 
> (i.e., no compositing window manager)?

idr, you aren't running a compositing wm, right?  All of us with
gnome-shell experience intermittent failures, with some tests showing up
more frequently than others.  I have some concerns about how dri2
manages its buffers in the presence of multiple clients that might
explain what's going wrong, but I haven't managed to write a testcase
for the bug despite trying.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://lists.freedesktop.org/archives/piglit/attachments/20120119/8568f647/attachment.pgp>