[Swfdec] swfdec tests

[Benjamin Otte]

(cross-posted to Cairo list, as it is relevant to you guys, too.
Please reply appropriately)

This is a somewhat complicated problem that we've been aware of for a
while, but noone has come up with a good solution so far, so it has
remained unsolved.

Here's the problem: The image tests work like this: We have a number
of Flash files and a reference image for each Flash file describing
the rendering of that Flash file. We then render the Flash file with
Swfdec using Cairo's image backend and then compare this rendering
with the reference image using the same simple diff code Cairo uses in
its test suite.

However, Cairo's image backend changes its output from time to time.
The current example you are providing is subtle gradient changes that
occured in the 1.5 release series of Cairo. So what we'e been doing
for the latest set of releases is upgrading the reference images to
match the output of the newer Cairo version.

Of course, this is a somewhat unfortunate situation. First of all, we
don't want to upgrade images every 6 months. Currently, this is still
doable as it's only ~170 images, but I expect that to grow up to
multiple thousands of images. Every change would require manual
inspection of all images that changed, quite a cumbersome task. Then
there is the issue with us supporting multiple versions of Cairo
(Swfdec should work with Cairo as old as 1.2 in its current state),
and it's unfortunate if a lot of people can't run the image tests when
they don't run the same Cairo minor version. Also, if new versions of
Cairo come out, it's likely that make check fails for older Swfdec
ersions, as they don't have the updated images. And of course our
testsuite doesn't yet check for the Cairo version when doing tests, so
it reports failures, which is probably what you are seeing in that
bug.

Last but not least I realized that I ahve no clue about guarantees
from the Cairo people that image backend output doesn't change in
micro versions or doesn't vary between different architectures, even
though I'd expect that to be the case.

I talked about this problem with Carl on IRC and he didn't seem to
have a solution for this problem either. It seems the Cairo people too
manually update all their reference images whenever they do subtle
changes.
Fortunately for them, it's a bit easier as they're pretty sure about
the Cairo version they test against. Carl indicated that this might
become an issue now that they rely on an external libpixman.

So what I'm aiming at with this mail is a solution to this problem
that is as nice as possible. If someone can come up with a nice one,
please tell me.
Otherwise I'm going to add code to restrict image tests to a specific
Cairo version. However, I'm definitely  not going to add that code to
the stable series, so if you want to fix that bug in Gentoo, you'll
have to restrict make check to Cairo 1.6 in your build scripts.

Cheers,
Benjamin



On Fri, Apr 18, 2008 at 2:42 PM, Nguyen Thai Ngoc Duy <pclouds at gmail.com> wrote:
> Hi,
>
>  I got some test failures [1] in test/image (text-field-color-transform
>  and gradient). Are those tests allowed to fail?
>
>  Thanks
>
>  [1] https://bugs.gentoo.org/show_bug.cgi?id=216284
>  --
>  Duy
>  _______________________________________________
>  Swfdec mailing list
>  Swfdec at lists.freedesktop.org
>  http://lists.freedesktop.org/mailman/listinfo/swfdec
>