[PATCH weston v0 00/11] Add screenshot-based testing using headless backend

[Bryce Harrington]

This is an initial draft of a set of patches by Derek and I to implement
headless rendering testing for Weston.  Posting it for comment now that
we have working proof of concept, but before anything's finalized.

The idea is to allow tests to render the desktop to memory using pixman,
and then compare the screen output to reference images to verify that
things got drawn correctly.  For now, this uses a pixel equivalence
checker (same code as in Cairo); eventually, intent is to provide some
fuzzy checks to account for slight, allowed differences in pixel colors.

A clipping mechanism is included to permit exclusion of areas of the
screen the test doesn't care about.  For example, the desktop clock's
time will vary from run to run, and thus differ from reference images so
you can clip out a subregion that excludes it.  Clipping also makes the
reference images a LOT smaller in git.

This patchset is intended to resolve the following tickets:

    https://bugs.freedesktop.org/show_bug.cgi?id=83981  # Done
    https://bugs.freedesktop.org/show_bug.cgi?id=83987  # Todo


In particular, remaining work for this series:

  * Deterministic background test pattern
  * Write test output & results into test output directories
  * Environment variables for reference image path, output directory,
    and any other settables
  * Move files_equal(), etc. to the weston-test API
  * Various FIXMEs
  * Update testing documentation

And follow-up work:
  * Non-pixel-perfect checking (e.g. slight color differences, alpha
    blending variations, etc.)
  * Deterministic pointer cursor (if any)
  * Implement rendering tests!



Bryce Harrington (6):
  Indicate headless compositor presence in config.h
  compositor: Document options for headless compositor
  Add a fadein test
  Add test reference directory and images for fadein
  tests: Add support for comparing output against reference images
  test:  Use only a clipped portion of screenshot for comparisons

Derek Foreman (5):
  backends: refactor transform string parsing
  compositor-headless: allow rendering with pixman
  compositor-headless: add support for transforms set on command line
  tests: Allow tests to use customized command line parameters
  tests: add screenshot recording to weston-test

 Makefile.am                     |  11 +++-
 configure.ac                    |   3 +
 protocol/wayland-test.xml       |   7 +++
 src/compositor-drm.c            |  30 ++-------
 src/compositor-headless.c       |  73 +++++++++++++++++++---
 src/compositor-rpi.c            |  43 ++-----------
 src/compositor-wayland.c        |  25 ++------
 src/compositor-x11.c            |  29 +--------
 src/compositor.c                |  49 +++++++++++++++
 src/compositor.h                |   6 ++
 tests/fadein-test.c             | 131 ++++++++++++++++++++++++++++++++++++++++
 tests/reference/fadein-00-0.png | Bin 0 -> 737 bytes
 tests/reference/fadein-01-0.png | Bin 0 -> 3453 bytes
 tests/reference/fadein-02-0.png | Bin 0 -> 3499 bytes
 tests/reference/fadein-03-0.png | Bin 0 -> 3461 bytes
 tests/reference/fadein-04-0.png | Bin 0 -> 3461 bytes
 tests/reference/fadein-05-0.png | Bin 0 -> 3461 bytes
 tests/weston-test-runner.c      |   8 +++
 tests/weston-test.c             |  87 ++++++++++++++++++++++++++
 tests/weston-tests-env          |   1 +
 20 files changed, 379 insertions(+), 124 deletions(-)
 create mode 100644 tests/fadein-test.c
 create mode 100644 tests/reference/fadein-00-0.png
 create mode 100644 tests/reference/fadein-01-0.png
 create mode 100644 tests/reference/fadein-02-0.png
 create mode 100644 tests/reference/fadein-03-0.png
 create mode 100644 tests/reference/fadein-04-0.png
 create mode 100644 tests/reference/fadein-05-0.png

-- 
1.9.1