[PATCH weston v3 00/15] Implement screenshot-based testing with the headless renderer

[Bryce Harrington]

Thought I'd try one last run at the football...

This series adds support for implementing test cases that can check
rendering output without needing connection to a physical output.  It
utilizes the pixman renderer in the headless backend to generate
screenshots at the request of the test client.

The test creates a shm buffer and passes it to the server via the
Weston-Test protocol ('capture_screenshot').  The server then renders
the display contents into the buffer and returns it as a response
('capture_screenshot_done').

The test then loads a corresponding reference PNG from disk using Cairo,
and then compares it with the captured screenshot.  Note that the
screenshot includes the current time in the desktop clock, which will of
course be different in the captured screenshot from the reference image.
So this checks a small clipped out section of each of the two images to
verify congruence.

By default, Wayland fades in the display and will show a patterned
background.  The former feature causes a black or nearly-black image to
be captured (the darkness of which may vary from system to system), and
the latter merely adds noise in our comparison, so both features are
disabled via a test-specific configuration .ini file.

[Update v2]
 * Refactor cairo out of the test client backend code entirely
   by utilizing the weston test surface structure to carry the
   specific data that the check_ routines need.
 * Drop refactoring redundant pixel copying code to shared
 * Take stride and bpp into account in check functions
 * In test case, set the bpp for the screenshot
 * Fix screenshot protocol description
 * Convert C++ style comments to C
 * Don't leak screenshot object
 * s/ec/compositor/
 * Instead of spinning on client_roundtrip, check for client's wl_display
 * Drop unneeded wl_surface in test case
 * Restore missing weston_log in idle_launch_client()
 * Drop unused test object from capture_screenshot
 * check_ function variable cleanup
 * Prefer tracking stride rather than bpp
 * Cleanup character calculation in check_ routines
 * Drop unnecessary data copying before write to disk
 * Split up test case code into several functions
 * Use screenshot_output_filename() so output files respect the
   WESTON_TEST_OUTPUT_PATH env var

[Update v3]
 * Stripped out support for handling dissimilar strides.  This probably
   will never come up in practice; if it does it can be handled then.
 * Improve code style in a few places.


Bryce Harrington (15):
  tests: Add error handling for system calls
  tests: Add client helper routines for output and reference filenames
  tests: Add an xmalloc helper function
  tests: Add surface checks
  tests: Support --config to enable tests to override config defaults
  protocol: Add test screenshot capability
  tests: Add screenshot recording capability to weston-test
  tests: Handle screenshot done event in weston-test
  tests: Add internal test for the weston test screenshot capability
  tests: Add write_surface_as_png() helper
  tests: Add create_screenshot_surface()
  tests: Add load_surface_from_png()
  tests: Add check_surfaces_geometry()
  tests: Add capture_screenshot_of_output()
  tests: Fix code style on path/filename routines

 Makefile.am                                |  23 ++-
 protocol/weston-test.xml                   |  16 ++
 tests/internal-screenshot-test.c           | 233 +++++++++++++++++++++++++++
 tests/internal-screenshot.ini              |   3 +
 tests/reference/internal-screenshot-00.png | Bin 0 -> 5149 bytes
 tests/weston-test-client-helper.c          | 164 +++++++++++++++++++
 tests/weston-test-client-helper.h          |  40 ++++-
 tests/weston-test.c                        | 243 +++++++++++++++++++++++++++++
 tests/weston-tests-env                     |  18 ++-
 9 files changed, 728 insertions(+), 12 deletions(-)
 create mode 100644 tests/internal-screenshot-test.c
 create mode 100644 tests/internal-screenshot.ini
 create mode 100644 tests/reference/internal-screenshot-00.png

-- 
1.9.1