[cairo] Automated testing of Cairo

[Carl Worth]

On Thu, 10 Aug 2006 15:33:22 -0700, Bryce Harrington wrote:
> Heh, you found the part that I was least satisfied with.  ;-)
> Yes, that area does require some work.
>
> For now, if you click on the run ID and go into the sysinfo dir you can
> find a summary page for each machine.  E.g.:

Ah, thanks for pointing that out. As long as the information exists
somewhere, that's all good.

> > 	1.2.3-c3c7068 (diff)
> >
> >    where the "1.2.3-c3c7068" links into gitweb and the "diff" is a
> >    link to the diff you currently provide.
>
> Hmm, I may have to punt on this one.

I'm a bit confused here as the current result I see for:

	2006-08-10 cairo-1.2.2
	c3c7068 master (diff)

seems exactly like what I was asking for. So thanks!

> > 3) The results all say "OK" now regardless of what failures
> >    exist. We're going to need to make that say something very
> >    different than "OK" for failures if this is going to be useful. ;-)
>
> Yes!  This was something I wanted to bring up.  Can you give me some
> sort of heuristic to use for differentiating between OK and BAD runs?

The most reliable indication of pass/failure is the return value of
each individual test, and that return value is already being examined
by the program that runs when you call "make test" or "make check" and
it prints PASS, FAIL, XPASS, or XFAIL as appropriate.

So the most reliable thing to look at is the output of "make test" or
"make check" looking for any lines beginning with FAIL. If you get any
of those, then the test run is not OK, (assuming we fix all the false
positives first---more on that below).

> What would you suggest we use to detect failures?  I see there are
> pass/fail counts in the html file - maybe something tied to that?

The HTML output is generated by grubbing through the log files, (which
were originally intended for giving humans more information after a
failure was reported). I would not recommend you then grub through its
output, as you'd be getting very far from the original failure reports
making it more likely that something could get missed somewhere,
(think a crashing test that makes the log file not get written
somehow, for example).

[snip discussion of win32 stuff]

> Hmm.  I would have to check with management on that.  OSDL engineering
> is pretty firmly FOSS, but if Cairo is providing and administering the
> machine, then in the sake of interoperability it might be possible.
> Can you get your hands on a machine and an admin for this?

Heh. I was asking only because I don't have any machines/admins, (nor
would I be able to come up with one personally nor through my employer
for similar "firmly FOSS" reasons. :-)

I see you're now talking to Vladimir, who is our current win32
maintainer and someone who would be most interested in seeing this
kind of automated testing happen. So hopefully something can be
determined through that discussion.

> Okay; give me a listing of the prerequisites to install in order to get
> all these things turned on, and I'll get it set up.  (We'll need to
> update the images for the various machines, so it's easiest if we can do
> all the deps at once.)

For the SVG and PDF backends you'll want rather recent versions of
librsvg and poppler. I'm afraid I don't know precisely which versions
are necessary to get no test failures. I believe most of the reference
images have been generated with both librsvg and poppler from recent
cvs checkouts, while other packages (such as ghostscript) have come
from Debian unstable or various versions of Fedora.

Other things that are necessary are freetype, (recent versions?), and
the Bitstream Vera font.

We've definitely done a very poor job of documenting all of these
dependencies and version that are required for getting the magic "all
tests give expected results" state. Hopefully, by going through a
round of this with you we can generate that list more precisely.

> nfs08:  GPL Ghostscript 8.50 (2005-12-31)
>         Input formats: PostScript PostScriptLevel1 PostScriptLevel2
>         PostScriptLevel3 PDF
>         Default output device: x11
> nfs09:  Not installed
> nfs11:  ESP Ghostscript 8.15.2 (2006-04-19)
>         Input formats: PostScript PostScriptLevel1 PostScriptLevel2
>         PostScriptLevel3 PDF
>         Default output device: bbox

OK. So my ghostscript is also "ESP Ghostscript 8.15.2 (2006-04-19)" so
I guess that's currently the "magic" version for the sake of the test
suite reference images. I can't make sense of the versions above to
understand if that is newer or older than GPL Ghostscript 8.50.

> > 6) The only current failure I see that isn't an obvious false positive
> >    like those described above is the failure of
> >    ft-text-vertical-layout which can be seen here:
> >
> > 	http://crucible.osdl.org/runs/1466/test_output/cairo-test/nfs11/test/
...
> Hopefully this has the info you need:
>
>    http://crucible.osdl.org/runs/1466/sysinfo/nfs11.1/

So the big different variable there is "gentoo" rather than
"debian". That could mean a lot of different things of
course. Debugging this one might be easiest for me once I can get
logged in to that machine and poke around.

> Okay.  There's probably going to be a variety of different issues to
> work through, but I've just now hooked them all up:
>
>   http://crucible.osdl.org/runs/cairo_branches.html
>
> amd01:  OK
> ppc01:  Fails during configure
> ita01:  Bunch of interesting issues during make check
> nfs12:  OK

So it would be helpful to have more details on those problems. Got any
now? Or maybe I can get them myself later.

> Hopefully as we develop more performance charting tools, we'll be able
> to apply them to Cairo too.

This all sounds great. My plan is to just have cairo/perf spit out raw
tables of numbers, and I was hoping that some automated charting would
just appear for us. Thanks in advance for anything you do to help make
that happen!

-Carl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.freedesktop.org/archives/cairo/attachments/20060816/dc379d43/attachment.pgp