[Pixman] [PATCH 1/2] Extend blitters test to first check known failure cases

[Siarhei Siamashka]

On Monday 12 April 2010, Soeren Sandmann wrote:
> Siarhei Siamashka <siarhei.siamashka at gmail.com> writes:
> > On Thu, Apr 8, 2010 at 2:39 AM, Søren Sandmann <sandmann at daimi.au.dk> 
wrote:
> > > From: Søren Sandmann Pedersen <ssp at redhat.com>
> > >
> > > This serves to make sure we don't accidentally reintroduce old bugs.
> >
> > The problem (or a feature) of the current tests is that crc32 can
> > change any time and such lists of known old failure cases of may be
> > invalidated any time in the future.
>
> Is this any different what we have now? If something changes, we have
> to update the CRC values. Adding a list of old failures just means
> there are more CRC values that would have to be updated, but this is a
> fairly mechanical change.

When not just pixman behavior, but the test itself changes (for example to
extend the list of image formats), then it is impossible to regenerate
the list of old failures because the same seed would result in a different
test being actually executed.

> > A proper solution would be to generate reproducible traces for the
> > failed testcases (record all the sequence of function calls, their
> > parameters and input data, plus store the expected output data).
>
> Sure, in fact a pixman-trace program similar to cairo-trace would be
> useful for a lot of things. I agree tests would ideally be done this
> way.

Even without making a fully featured cairo-trace counterpart, just extending
the current blitters-test with logging of all the operations (and spitting
out C source or script) can help to generate testcases for checking old
failures.

> However, there is still value in a simple table of regressions that
> have failed previously. For example the MMX bug is the kind of thing
> that could easily be reintroduced in other backends and not be found
> in the real world since it only shows up with solid alpha masks that
> have an RGB format and a non-zero color component.

The current bruteforce-alike testing has a high probability of detecting both
old and potentially new issues anyway. It's only a matter of running it
periodically on all the supported systems and configurations.

An extra simple table is going to only quickly detect the problems which
are already exercised by the same test a bit later. So it looks redundant
in the current form.

The real problem is checking for old failures, which are really hard to
reproduce. Something like this old bug:
https://bugs.freedesktop.org/show_bug.cgi?id=23934
It required 59979688 test iterations to be detected, which needs ~30x more
time to run than the default blitters-test from 'make check'.

> > I have a work-in-progress branch here intended to improve pixman
> > regression testing (and messes up all the checksums):
> > http://cgit.freedesktop.org/~siamashka/pixman/log/?h=tests-refactoring
>
> That looks mostly good, especially the fuzzer_main() function. 

> I don't see how it improves *regression* testing, though. (Regression
> testing meaning making sure that old bugs are not reintroduced).

Well, in a broader sense, regression testing means preventing regressions
(so that the new version does not become worse, or at least does not fail
the tests which worked fine with the previous one).

> > The current problems if the regression tests include:
> > 1. missing a8b8g8r8 format for mask in blitters-test
> > 2. scaling-test coverage is actually not good enough (it's
> > embarassing, but it did not take into account that the random number
> > generator has a limited range for generated values).
> >
> > Also this branch tries to utilize multiple CPU cores via OpenMP to
> > speed up the tests. This seems to be a very simple change, but works
> > fine and is supposed to be portable.
>
> Will this just automatically work with GCC or do you have to compile
> it in a special way?

Yes, with "-fopenmp" option added to CFLAGS, which can be also detected 
automatically by configure. Though there is still a risk of encountering
systems with OpenMP supposedly supported, but broken when used with pixman
tests.

-- 
Best regards,
Siarhei Siamashka