[cairo] [cairo-commit] Giant revamp of test suite,

[Carl Worth]

On Fri, 2008-10-31 at 05:31 -0700, Chris Wilson wrote:
> commit e90073f7ddc6f461a935bc360c409b04f1fe9f74
> Author: Chris Wilson <chris at chris-wilson.co.uk>
> Date:   Wed Sep 3 16:38:03 2008 +0100
>
>    [test] Build test suite into single binary.
...
> +The test suite is built as a single binary, which allows you to choose
> +individual or categories of tests to run. For example, if you want to
> +run all text related tests you can use:
> +    ./cairo-test-suite text
> +Or if you want to check the current status of known failures:
> +    ./cairo-test-suite XFAIL

Thanks for this big change, Chris! It should be quite nice.

I'm sure I will run into some minor frustrations as I use this that will
be easily fixed. Trying to predict some of these (though maybe I should
actually just try it for a little while first):

* I know I often want to just run a single test and it's been nice to
just be able to use ./[TAB] to get tab-completion for that. I suppose I
still have that now, but I just have to type "./cairo-test-suite [TAB]"
instead. So:

	The documentation above says that I can choose an individual test,
	but I did miss that at first. Maybe add an example such as:

	./cairo-test-suite fill-and-stroke

	Will ./c[TAB] complete unambiguously to ./cairo-test-suite. I
	suppose it should since we've dropped so many binaries from this
	directory. So maybe this usage will be fine. Otherwise, I was maybe
	going to ask for a million little shell scripts for running individual
	tests (hideous I know).

> +This binary should be backwards-compatible with all library versions,
> +allowing you to compare current versus past behaviour for any test.

Could you expand on that a bit here in the documentation? I know that
for a while the test suite has been careful to ensure that it is linked
with the compiled-but-uninstalled local library, which is almost always
what is wanted. But it would be quite nice to run against other
installed versions as well. How should we do that exactly? Do we need to
do "chrpath -r" tricks to avoid the fact that libtool has used rpath to
force linkage with the local copy? I'd like to hear more about what
you've come up with for this use case.

> +To limit the tests run, use the CAIRO_TESTS environment variable, which
> +should be a space-separated list of tests to run.  For example:
>  
> -  make test TARGETS=image,ps3 TESTS="zero-alpha"
> +  CAIRO_TEST="zero-alpha" make test TARGETS=image,ps

That should be "CAIRO_TESTS=" I assume.

But more significantly, I wonder about the two different approaches we
are documenting here for running the test suite, (both cairo-test-suite
and make-test). That seems unnecessarily confusing.

It was this invocation of:

	CAIRO_TEST="fill-and-stroke" make test

that I imagined having to type that made me first start thinking about
shell scripts. That invocation would be really painful, (and would
presumably prevent tab-completion). The ./cairo-test-suite thing seems
much better, and I imagine we could even do things like:

	./cairo-test-suite --targets=image,ps

and then we could even go crazy and do custom bash-completion support
for options like that.

Another thing to think about is providing support to install the
cairo-test-suite so that people could use it on their systems separate
from a source tree of cairo, (presumably with reference images
in /usr/share/cairo-test-suite or whatever). I'd like to do something
similar for cairo-perf and cairo-trace, so we should probably just plan
to tackle all similar issues together.

Keep up the great work, Chris!

-Carl

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.cairographics.org/archives/cairo/attachments/20081031/034eea5a/attachment-0001.pgp