[cairo] Cairo testing for LSB

[Stew Benedict]

On Wed, 13 Aug 2008, Carl Worth wrote:

> There are various possible issues. One of the first, most essential
> pieces of missing information from you is which backends are failing. A
> single FAIL on a test without saying which backend failed really doesn't
> say much. Newer cairo versions are nice about putting a little report
> together at the end of the run showing a per-test list of which backends
> failed, and a per-backend list of which tests failed.
> 
> Also, the test suite generates a .html file with links to all the
> generated images, (output, reference, and diff). We'll really need to
> see those to be able to say what's going on.
> 

I did go back and add the detail info out of the runs (essentially what is 
in the .log file, that I also capture with TET.  I didn't find any html 
file, but maybe the 1.0.2 tests didn't have this feature or my "port" lost 
it. I'll look again at the source tarball.

If I was to move to a newer version of the test suite, with the 
additional features, how do you think things will work out testing older 
cairo versions? If need be, I can of course filter out tests that would be 
expected to fail against >= 1.0.2 (we're held back at this version due to 
where the enterprise distros are at).

> Some of the things that can cause lots of "failures" even when things
> are going well include:
> 
> * Differences in details of how X servers (or X server drivers)
> rasterize the graphics
> 
> * Different fonts being used in the tests than in the reference images
> 
> * Differences in details of how the font rasterizer, (freetype, say),
> rasterizes text glyphs.
> 
> We're always working to make the test suite more resilient to these
> issues, (for example, we're bundling more and more of the fonts directly
> with the suite). But it's not something we'll ever be entirely free of.
> We could also do a much better job of documenting exactly what needs to
> be done to be able to get a "perfect" run of the test suite. (And you
> can definitely help us develop such documentation as you go through this
> process).
> 

This is great. I can control the process to some degree using our own Xvfb 
and fonts. Freetype differences across the distros could become 
"interesting".

-- 
Stew Benedict
The Linux Foundation