[cairo] Automated testing of Cairo

[Bryce Harrington]

On Wed, Aug 16, 2006 at 01:49:36PM -0700, Carl Worth wrote:
> On Wed, 16 Aug 2006 13:00:54 -0700, Bryce Harrington wrote:
> >
> > Ah cool.  The bit I was unsure about was making it report 1.2.3 instead
> > of 1.2.2, but if you're okay with it showing 1.2.2 in the above, that's
> > great.  :-)
> 
> Oh, that. No, I still don't like the version being wrong. ;-) 
> OK. Let's leave it as an in-the-future thing. But it would be nice to
> have fixed.

I poked through things yesterday and think I understand what the problem
is that I had been worried about, but there's a few other issues (like
that I'm not pulling down your git patches correctly) that will have to
get fixed first.  I think we may have it fixed by the end of the month.

> It doesn't really matter for now since we don't yet have _any_
> versions of 1.2 on that page that are showing no failures. ;-)
> 
> >   http://crucible.osdl.org/runs/cairo_branches.html

Note, since we're not doing branch testing, I'm going to rename the link
to this:

   http://crucible.osdl.org/runs/cairo_report.html

Both links will continue to work, but please use this latter one; I will
eventually remove the first one.

> > The numbers shown are (fails / fails+passes).
> 
> This looks good. Particularly, now since I had you count only separate
> tests and not tests X backends X 2 content X 2 offsets. The only funny
> things I still see is that the count of total tests is differing
> across the systems even for the same version of cairo. A very small
> difference, (5 tests or fewer), is actually expected if there are
> different backends enabled on some systems. This is because there are
> a few backend-specific tests that won't be run at all if the backend
> isn't enabled. So 88 tests without xlib and 89 tests with xlib looks
> fine.
> 
> But 176 tests instead of 88 tests looks like some miscounting is
> happening somewhere.

Sorry, that was just a result of me manually re-running stuff on those
machines without clearing the prior time's log first.

I've just now re-run another test of that patch to give a clean set of
test results.  It's showing 88 or 89 tests across the board.

Also, I've upgraded many of the packages you'd suggested, so this is now
including SVG and xlib results for most systems.  (The itanium is
erroring during the xorg compile; I'll look into that more another day.)

> > If you have other ideas on how to make the display more immediately
> > useful, let me know and I'll see what I can do.
> 
> Maybe it would help to add the total test count OK cells too. The idea
> there is that if we transition 89 tests to 1 failure/90 tests that's
> quite different from transitioning from 89 tests to 1 failure/89
> tests.

Done.  Good idea.

> > Well, here is what's in gentoo for x86 stable:
> >
> >  gnome-base/librsvg-2.12.7  USE="zlib -debug -doc -gnome -nsplugin"
> 
> That might be too old of a version of librsvg, (though we won't really
> know until we try as I don't see any svg backend testing happening
> yet). Emmanuel Pacaud is cairo's SVG backend maintainer and he
> confirms that librsvg 2.14 is sufficient for testing cairo.

Okay, runs 1516 and 1517 are with this version of librsvg and include
SVG results.  I'm going to assume this librsvg is going to be too old.
Gentoo unstable has this version available, which I'll upgrade to:

 dev-libs/libcroco-0.6.1 [0.6.0] USE="-debug"
 gnome-base/librsvg-2.14.3 [2.12.7] USE="zlib -debug -doc -gnome"

Once the systems have finished upgrading to the unstable librsvg I'll
run another set of tests, then try upgrading poppler and see what that
does.

> > The logs/ directory contains the details.  For kernel stuff we break
> > these up into separate files for configure, make, make check, etc. but
> > for cairo its all in one file.  For convenience, I've now also
> > hyperlinked this log file from the cairo_branch.html page.
> 
> I mentioned this on irc, but there appears to be a web server problem
> in that the log files are being served with a mime type of a gzipped
> tar file rather than plain text, so they're difficult to browse on the
> web. Presumably some change to apache configuration would fix
> that. Otherwise, perhaps renaming the files to not have a .log
> extension rather than .tar.gz.log would workaround the problem.

Fixed.  Apache just needed a FileType for .log files
 
> I took a look at the ppc configure failure and the relevant output
> appears to be:
> 
> 	checking for cairo's PNG backend...
> 	configure: WARNING: Could not find libpng in the pkg-config search path
> 	checking whether cairo's PNG backend could be enabled... no
> 	configure: error: requested PNG backend could not be enabled
> 	CONFIGURE STATUS: 1
> 
> Is the libpng devel package simply not installed on that machine?

It's installed now, and the ppc is returning results now.

Bryce