[PATCH weston 00/10] Implement screenshot-based testing with the headless renderer

[Pekka Paalanen]

On Sat, 9 May 2015 15:01:13 +0100
Daniel Stone <daniel at fooishbar.org> wrote:

> Hi Bryce,
> 
> On 7 May 2015 at 01:44, Bryce Harrington <bryce at osg.samsung.com> wrote:
> > This series adds support for implementing test cases that can check
> > rendering output without needing connection to a physical output.  It
> > utilizes the pixman renderer in the headless backend to generate
> > screenshots at the request of the test client.
> >
> > The test creates a shm buffer and passes it to the server via the
> > Weston-Test protocol ('capture_screenshot').  The server then renders
> > the display contents into the buffer and returns it as a response
> > ('capture_screenshot_done').
> >
> > The test then loads a corresponding reference PNG from disk as a cairo
> > surface, and then compares it with the captured screenshot.  Note that
> > the screenshot includes the current time in the desktop clock, which
> > will of course be different in the captured screenshot from the
> > reference image.  So this checks a small clipped out section of each of
> > the two images to verify congruence.
> >
> > By default, Wayland fades in the display and will show a patterned
> > background.  The former feature causes a black or nearly-black image to
> > be captured (the darkness of which may vary from system to system), and
> > the latter merely adds noise in our comparison, so both features are
> > disabled via a test-specific configuration .ini file.
> 
> Modulo the comments I had earlier (and loads of C++ comments in
> 09/10), these do look good to me.
> 
> Thanks a lot for doing them - they're very useful, and I'd like to see
> them get into 1.8 if possible.
> 
> They're certainly not perfect, and I think show up a lot of the
> limitations of our current test 'infrastructure', but IMO that should
> in no way block these tests. On the contrary, it gives us more data to
> build our test framework on: it will make the conversion a little more
> painful, but I think be a good guide for its development.

Hi Bryce,

the approach and design look good.

For patches 1, 2, 3, 5, 8, 
Reviewed-by: Pekka Paalanen <pekka.paalanen at collabora.co.uk>

Patches 4, 6, 7 need fixing. Patch 9 should get minor changes. Patch 10
would be better dropped as it is, since all that should be replaced by
just a pixman_image_composite32() call. If there really was a reason to
hand-code all that copying, I've long forgot it.

Should weston-tests-env also be setting WESTON_TEST_OUTPUT_PATH and
WESTON_TEST_REFERENCE_PATH?

Right now by default, an out-of-tree 'make check' will fail for not
finding the reference image.

Did you notice, that 'clientside-screenshot.png' ends up all black, if
you do 'make check BACKEND=x11-backend.so'?
Wonder why it does that, you're not intending to do anything
backend-specific. Except the server command line arguments?

Jon, this series adds a test type that would be useful to run in several
different setups: backends x renderers. I think that's something you
could take into account.


Thanks,
pq