[Pixman] [PATCH 1/2] Extend blitters test to first check known failure cases

[Soeren Sandmann]

Siarhei Siamashka <siarhei.siamashka at gmail.com> writes:

> On Thu, Apr 8, 2010 at 2:39 AM, Søren Sandmann <sandmann at daimi.au.dk> wrote:
> > From: Søren Sandmann Pedersen <ssp at redhat.com>
> >
> > This serves to make sure we don't accidentally reintroduce old bugs.
> 
> The problem (or a feature) of the current tests is that crc32 can
> change any time and such lists of known old failure cases of may be
> invalidated any time in the future.

Is this any different what we have now? If something changes, we have
to update the CRC values. Adding a list of old failures just means
there are more CRC values that would have to be updated, but this is a
fairly mechanical change.

> A proper solution would be to generate reproducible traces for the
> failed testcases (record all the sequence of function calls, their
> parameters and input data, plus store the expected output data).

Sure, in fact a pixman-trace program similar to cairo-trace would be
useful for a lot of things. I agree tests would ideally be done this
way.

However, there is still value in a simple table of regressions that
have failed previously. For example the MMX bug is the kind of thing
that could easily be reintroduced in other backends and not be found
in the real world since it only shows up with solid alpha masks that
have an RGB format and a non-zero color component.

> I have a work-in-progress branch here intended to improve pixman
> regression testing (and messes up all the checksums):
> http://cgit.freedesktop.org/~siamashka/pixman/log/?h=tests-refactoring

That looks mostly good, especially the fuzzer_main() function. I don't
see how it improves *regression* testing, though. (Regression testing
meaning making sure that old bugs are not reintroduced).

> The current problems if the regression tests include:
> 1. missing a8b8g8r8 format for mask in blitters-test
> 2. scaling-test coverage is actually not good enough (it's
> embarassing, but it did not take into account that the random number
> generator has a limited range for generated values).
> 
> Also this branch tries to utilize multiple CPU cores via OpenMP to
> speed up the tests. This seems to be a very simple change, but works
> fine and is supposed to be portable.

Will this just automatically work with GCC or do you have to compile
it in a special way?


Soren