[cairo] Cairo testing for LSB

[Carl Worth]

On Wed, 2008-08-13 at 06:49 -0400, Stew Benedict wrote:
> Hi everyone,

Hi Stew,

> My name is Stew Benedict and I'm working for the Linux Foundation on LSB.

Welcome to cairo, and we're definitely glad to have more help examining
our releases and testing them. So thanks!


> We're including libcairo in LSB4.0 and I've been looking at incorporating 
> the unit tests from the tarball as a runtime test for the distributions. 
> I've been able to patch the tests for TET output (framework LSB uses) and 
> can build/run the tests OK, but I'm seeing rather poor results as I test 
> against newer versions of cairo (we're basing LSB off of 1.0.2, which is 
> what SLES10 has). 
> 
> There's a page here of what I've run into, if you have any insights or 
> suggestions it would be greatly appreciated.
>
> X server in all cases is an LSB build of Xvfb, render extension 
> enabed, bundled with 75dpi, 100dpi, misc fonts from xorg.
> 
> http://www.linuxfoundation.org/en/CairoTesting40

There are various possible issues. One of the first, most essential
pieces of missing information from you is which backends are failing. A
single FAIL on a test without saying which backend failed really doesn't
say much. Newer cairo versions are nice about putting a little report
together at the end of the run showing a per-test list of which backends
failed, and a per-backend list of which tests failed.

Also, the test suite generates a .html file with links to all the
generated images, (output, reference, and diff). We'll really need to
see those to be able to say what's going on.

Some of the things that can cause lots of "failures" even when things
are going well include:

* Differences in details of how X servers (or X server drivers)
rasterize the graphics

* Different fonts being used in the tests than in the reference images

* Differences in details of how the font rasterizer, (freetype, say),
rasterizes text glyphs.

We're always working to make the test suite more resilient to these
issues, (for example, we're bundling more and more of the fonts directly
with the suite). But it's not something we'll ever be entirely free of.
We could also do a much better job of documenting exactly what needs to
be done to be able to get a "perfect" run of the test suite. (And you
can definitely help us develop such documentation as you go through this
process).

Thanks,

-Carl