[PATCH weston v3 00/15] Implement screenshot-based testing with the headless renderer

[Derek Foreman]

With the tiny stride change mentioned on IRC, which, for those that
weren't present, was basically little more than:
if (stride != bpp * reference->width) FAIL
in load_surface_from_png()

for the series:
Reviewed-By: Derek Foreman <derekf at osg.samsung.com>


On 21/05/15 03:39 AM, Bryce Harrington wrote:
> Thought I'd try one last run at the football...
> 
> This series adds support for implementing test cases that can check
> rendering output without needing connection to a physical output.  It
> utilizes the pixman renderer in the headless backend to generate
> screenshots at the request of the test client.
> 
> The test creates a shm buffer and passes it to the server via the
> Weston-Test protocol ('capture_screenshot').  The server then renders
> the display contents into the buffer and returns it as a response
> ('capture_screenshot_done').
> 
> The test then loads a corresponding reference PNG from disk using Cairo,
> and then compares it with the captured screenshot.  Note that the
> screenshot includes the current time in the desktop clock, which will of
> course be different in the captured screenshot from the reference image.
> So this checks a small clipped out section of each of the two images to
> verify congruence.
> 
> By default, Wayland fades in the display and will show a patterned
> background.  The former feature causes a black or nearly-black image to
> be captured (the darkness of which may vary from system to system), and
> the latter merely adds noise in our comparison, so both features are
> disabled via a test-specific configuration .ini file.
> 
> [Update v2]
>  * Refactor cairo out of the test client backend code entirely
>    by utilizing the weston test surface structure to carry the
>    specific data that the check_ routines need.
>  * Drop refactoring redundant pixel copying code to shared
>  * Take stride and bpp into account in check functions
>  * In test case, set the bpp for the screenshot
>  * Fix screenshot protocol description
>  * Convert C++ style comments to C
>  * Don't leak screenshot object
>  * s/ec/compositor/
>  * Instead of spinning on client_roundtrip, check for client's wl_display
>  * Drop unneeded wl_surface in test case
>  * Restore missing weston_log in idle_launch_client()
>  * Drop unused test object from capture_screenshot
>  * check_ function variable cleanup
>  * Prefer tracking stride rather than bpp
>  * Cleanup character calculation in check_ routines
>  * Drop unnecessary data copying before write to disk
>  * Split up test case code into several functions
>  * Use screenshot_output_filename() so output files respect the
>    WESTON_TEST_OUTPUT_PATH env var
> 
> [Update v3]
>  * Stripped out support for handling dissimilar strides.  This probably
>    will never come up in practice; if it does it can be handled then.
>  * Improve code style in a few places.
> 
> 
> Bryce Harrington (15):
>   tests: Add error handling for system calls
>   tests: Add client helper routines for output and reference filenames
>   tests: Add an xmalloc helper function
>   tests: Add surface checks
>   tests: Support --config to enable tests to override config defaults
>   protocol: Add test screenshot capability
>   tests: Add screenshot recording capability to weston-test
>   tests: Handle screenshot done event in weston-test
>   tests: Add internal test for the weston test screenshot capability
>   tests: Add write_surface_as_png() helper
>   tests: Add create_screenshot_surface()
>   tests: Add load_surface_from_png()
>   tests: Add check_surfaces_geometry()
>   tests: Add capture_screenshot_of_output()
>   tests: Fix code style on path/filename routines
> 
>  Makefile.am                                |  23 ++-
>  protocol/weston-test.xml                   |  16 ++
>  tests/internal-screenshot-test.c           | 233 +++++++++++++++++++++++++++
>  tests/internal-screenshot.ini              |   3 +
>  tests/reference/internal-screenshot-00.png | Bin 0 -> 5149 bytes
>  tests/weston-test-client-helper.c          | 164 +++++++++++++++++++
>  tests/weston-test-client-helper.h          |  40 ++++-
>  tests/weston-test.c                        | 243 +++++++++++++++++++++++++++++
>  tests/weston-tests-env                     |  18 ++-
>  9 files changed, 728 insertions(+), 12 deletions(-)
>  create mode 100644 tests/internal-screenshot-test.c
>  create mode 100644 tests/internal-screenshot.ini
>  create mode 100644 tests/reference/internal-screenshot-00.png
>