[Pixman] Testing (Re: [PATCH 3/3] ARMv6: Add fast path for over_n_8888_8888_ca)

[Pekka Paalanen]

On Mon, 7 Apr 2014 10:50:03 +0300
Siarhei Siamashka <siarhei.siamashka at gmail.com> wrote:

> On Fri, 4 Apr 2014 10:28:05 +0300
> Pekka Paalanen <ppaalanen at gmail.com> wrote:
> 
> > Could someone point me to a document describing how one uses these
> > testing tools properly? Hopefully it would answer all my questions
> > below.
> 
> Unfortunately the only basic document describing how to use this
> particular testing tool is printed if you run the fuzzer-find-diff.pl
> script without any arguments. And also a comment in the code for the
> fuzzer_test_main() function:
>     http://cgit.freedesktop.org/pixman/tree/test/utils.c?id=pixman-0.32.4#n670
> But you have found all of this already and I have no additional links
> or documents to share. Sorry about this. Though google search may also
> have some hits in the pixman mailing list.
> 
> The documentation clearly needs improvements. Your feedback is valuable
> and helps to identify the gaps in it.

...

> > OTOH, I see that fuzzer_test_main() takes a checksum as an argument.
> > How do you determine what the correct checksum should be?
> > After reading the big comment on fuzzer_test_main() and the usage of
> > fuzzer-find-diff.pl, I'm getting the hunch, that the procedure would be
> > something like this:
> 
> We just assume that the current pixman code is correct and run the
> test. It naturally fails, but reports something like this:
> "expected XXXXXXXX, got YYYYYYYY". Then we take this YYYYYYYY checksum
> and hardcode it in the sources of the test. The assumption is that this
> test is going to still produce YYYYYYYY checksum on any platform
> regardless of what optimized fast paths they have or don't have.

Ok, I was more interested in how you can assume that the current pixman
code really is correct, especially when adding new (fuzzer) tests,
since presumably the reason to add more tests is because something was
not tested well enough before.

But I guess there isn't really a way to make sure of that before-hand.
You assume the generic code is correct, ensure that it's output stays
the same, and look into it only if someone reports a bug with a
specific operation.

> Please note, that this is only one type of the tests in 'make check'.
> This approach does not really work well for floating point fast paths
> because we can't expect deterministic pixel perfect results. The other
> types of tests exist too.
> 
> Anyway, what you are describing below is just the procedure for
> narrowing the test failure to a single problematic seed using the
> fuzzer-find-diff.pl script:
> 
> > - compile pixman without optimizations producing statically linked
> >   blitter-test (how?), rename it to blitters-test.generic
> 
> Yes, you just use "--disable-shared" option for pixman configure.
> That's also a hint given by help message in the fuzzer-find-diff.pl
> script.
> 
> Or even "--enable-static-testprogs --disable-gtk --disable-libpng" to
> statically link everything including libc. This may be useful if you
> want to run this binary in android or with qemu-user.
> 
> > - compile pixman with optimizations producing statically linked
> >   blitter-test (how?), rename it to blitters-test.armv6 (on rpi)
> 
> You compile one binary configured as "--disable-arm-simd". And another
> one with arm-simd (armv6) optimizations still in place.
> 
> If you run the test on a high-end ARM hardware, "--disable-arm-neon"
> configure option is also needed to prevent the NEON fast paths from
> getting in the way.
> 
> Also in your case of having extremely slow Raspberry Pi hardware, it
> may be even beneficial to run the reference binary on your x86 box.
> The fuzzer-find-diff.pl script contains an example of making use of
> ssh to run binaries on different machines.
> 
> > - run with fuzzer-find-diff.pl for the predetermined number of rounds
> > - if no differences found, take the final checksum (from where?) and
> >   hardcode it in the fuzzer_test_main() call.
> 
> This is not needed. At least not for anything related to 'make check'.
> 
> > And normally that would be done only by the maintainers, or when
> > someone adds a new fuzzer test. Is that right?
> 
> Yes. If the behaviour of pixman changes (for a good reason and in the
> same way on all platforms), then the checksums are updated for these
> tests.
> 
> > For building the generic version, or whatever version is the gold
> > standard, should I use all the --disable switches mentioned
> > in ./configure --help?
> 
> Only "--disable-arm-simd"/"--disable-arm-neon" options are important
> for ARM here. But adding extra --disable switches will not do any harm
> either.

I wonder if it would be nice to have a separate configure switch to
force all the necessary configure options to produce the gold standard
build. Or maybe just a script to call configure with the right options?

> > And 'make check' only runs the whatever was built and checks just
> > against the hardcoded checksum?
> 
> Yes.
> 

Thank you for the excellent explanation!
- pq