[cairo] Automated testing of Cairo

[Bryce Harrington]

On Thu, Aug 10, 2006 at 02:55:17PM -0700, Vladimir Vukicevic wrote:
> On 8/10/06, Bryce Harrington <bryce at osdl.org> wrote:
> >Heya all,
> >
> >I'm happy to be able to announce that OSDL will be supporting Cairo by
> >providing automated testing on various platforms.
> 
> I see that you mention adding non-linux hardware; do you guys have any
> win32 infrastructure in place, or could you run win32 on your existing
> hardware?

We don't have any Win32 infrastructure.  OSDL is pretty firm about
focusing on Linux and Open Source exclusively - we're chartered this
way.  In fact it gets quite controversial with the community if we
appear to be doing anything that could be perceived to support
proprietary stuff.  For instance, we've gotten flamed in the past when
our marketing folks create PDFs on Macs using Adobe products.  ;-)

But I think we could get by with hosting a separate Win32 or OSX box
that someone else provides, and that is set up and administered by
non-OSDL people.  I'd need to run it by management and our IT staff.

The test system requires that the test machines be able to establish an
NFS mount and run a cronned perl script.  I would guess OSX can do this
no prob; Win32 might require a bit of adaptation.

> If not, we can probably run win32 and MacOSX performance
> tests (once we have a good suite) on our test boxes and push the
> results to a single location for display.

Yup.  Also, the test harness I use is itself all open source, and should
be feasible for you to set up and run on your own systems.  In fact,
since you won't need to reboot/reimage the machines or futz with the
kernel, you'd be able to avoid all the tricky stuff (systemimager,
PXELinux, power management, etc.)

> Also, for the performance tests, they will be extremeley sensitive to
> X version and drivers on Linux; ideally, the most useful numbers would
> be with Intel, Nvidia, and ATI hardware, using both open and
> proprietary drivers; that explodes the configuration matrix
> significantly though.  However, if you guys are virtualizing all of
> this, I'm not sure how the X perf numbers will relate to real-world
> machines (we're having this same problem with Mozilla's performance
> testing; our tinderboxes tend to be server-class machines, so testing
> rendering performance on them has misled us a number of times).

Yes, this is something I've been quite curious about.  Carl said even
just testing with server-class machines will provide useful info, but
I would love to know if there are ways we can also include X in the
testing.

We had looked into doing direct xorg testing previously, however it
seemed that the right thing to test was varying video cards, but we
didn't have budget or vendor participation to get the cards, so instead
got into storage driver testing (thanks to support from HP and NetApp).

But, if someone would like to take a field trip out here and BYO video
cards, we've got a bunch of cast-off desktops that I could probably talk
IT into letting us repurpose for Cairo / video driver testing.  Oregon's
beautiful right now, and we've got plenty of cube space and free 
wi-fi...  :-)

Bryce