[PATCH weston 2/2] internal-screenshot-test: Fix test so it doesn't expect shell surfaces

[Bryce Harrington]

On Tue, May 26, 2015 at 12:10:07PM -0500, Derek Foreman wrote:
> > 
> > Since we are procedurally generating these particular reference images,
> > I would not save them as PNGs in git to avoid growing the size.

PNGs which are procedurally generated potentially can have great
compression ratios, so this *might* not be that big of a deal in
practice.  E.g. where you can use large rectangular areas of just pure
black, that will get compressed extremely to a few bytes.

git appears to handle binary files a lot nicer than older version
control systems which would e.g. uuencode them.  The only thing I think
is a problem is it doesn't do well with binary diffs, but that's only an
issue if the binary file gets updated a lot; which for reference images
shouldn't be happening (and if it is, then perhaps is a sign to rethink
our test design...)

> > OTOH, saving even generated reference images as PNG in git has the
> > benefit that it will expose also (unintentional) changes in the
> > generator. But, that could be protected against by using a CRC (see
> > e.g. Pixman).

I am also a bit worried that generating the reference images might hide
problems, or might bring more false positive issues.

Size of the distributed weston tarball is less of a worry to me - it's a
reference compositor, so should not bother folks if it were to be full
of reference images.  Indeed there could be an argument made that
perhaps someday other compositor writers might like to look at the
reference images manually, in writing their own implementations of
weston's tests.

Besides, there's simpler way to save a lot of space.  Read on...

> I think we'd need a way to compare a reference image to part of a
> screenshot (sizes don't match but clip sizes do?)  This is similar if
> not exactly what you suggested previously about fixing the screenshot
> test, but I took a shortcut to save time

Fwiw, and this may be veering off-thread a bit, but I had thought of
storing only the clipped portion of references.  I decided not to bother
with that, for the following reasons:

 * Simplifies updating the reference image if you can just cp the output
   image after checking that it's correct.

 * Potentially, multiple tests may be able to reuse the same reference
   image file, but to do so may need to examine different clips.

 * As mentioned above, if you set the background to a solid color, this
   enables the PNG algorithms to compress more tightly.  If the client
   can also be a solid color, then the PNG files end up being small
   even unclipped.

 * If the filesize is an issue, and clipping it down saves significant
   space, why not just use a smaller rendering surface to begin with?
   This is why I set the test to 320x240.

This last point especially could be employed if we are indeed worried
about tarball size.  I picked 320x240 as a small resolution that still
gives ample flexibility for doing window operations and such.  But a lot
of our tests (including internal-screenshot) could probably do what they
need to do in 100x100 or less, and bring reference image file sizes down
by a factor of maybe 3 or more.

Since the tests can control what resolution weston is run at, if there
are specific tests needing larger resolutions they can just configure
it.

Anyway, that was my line of thinking...  Just keep the reference images
as simple as possible and straightforward for others to refer to easily.
If we're worried about the size impact they have on the tarball, then
just shrink the tests' display sizes.

Bryce