[Pixman] [PATCH] test: Change composite so that it tests randomly generated images

[Siarhei Siamashka]

On Sunday 07 March 2010, Søren Sandmann wrote:
> Previously this test would try to exhaustively test all combinations
> of formats and operators, which meant that it would take years to run.
>
> Instead, generate random images and test those. The random seed is
> based on time(), so we will get different tests every time it is run.
> Whenever the test fails, it will print what the value of lcg_seed was
> when the test began.
>
> This patch also adds a table of random seeds that have failed at some
> point. These seeds are run first before the random testing begins.
> Whenever this tests fails, you are supposed to add the seed that
> failed to the table, so that we can track that the bug doesn't get
> reintroduced.

I don't quite like any nondeterministic random factor in the standard 
regression tests. Preferably the results of such tests should be
reproducible from run to run, even if they are not perfect and do not
provide full coverage.

This is quite important for having a clearly defined formal patch submission
process (Before submitting a patch, one needs to make sure that the regression
tests pass. If they don't pass, the problem has to be investigated and patch
fixed or regression tests updated if needed).

With the randomness in the tests, patch contributor may end up in different
confusing situations:
- regression test fails for him, even if his patch is fine (if the problem was
introduced by somebody else earlier)
- regression test passes for him, but fails for the others later (due to the
bug in the patch). In this case it would be hard to say if the contributor did
proper job running the regression tests in the first place.

So I'm in favor of having a deterministic set of seed values for a standard
regression test and a special separate mode for finding new problematic seed
values.


Also the original test which tries all the possible combinations is a good
thing to have. Just to be sure that all the corner cases are checked. While
random test asymptotically gets closer to 100% coverage when run longer, it
may never reach it.

-- 
Best regards,
Siarhei Siamashka
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.freedesktop.org/archives/pixman/attachments/20100310/49ecd3c1/attachment.pgp>