Release blocker (Re: [PATCH weston v3 00/15] Implement screenshot-based testing with the headless renderer)

[Pekka Paalanen]

On Thu, 21 May 2015 15:43:18 -0700
Bryce Harrington <bryce at osg.samsung.com> wrote:

> On Thu, May 21, 2015 at 04:44:41PM -0500, Derek Foreman wrote:
> > With the tiny stride change mentioned on IRC, which, for those that
> > weren't present, was basically little more than:
> > if (stride != bpp * reference->width) FAIL
> > in load_surface_from_png()
> > 
> > for the series:
> > Reviewed-By: Derek Foreman <derekf at osg.samsung.com>
> 
> Thanks, landed (including the additional stride change):
>    f556ebe..0ccf8e2  master -> master

Hi,

with the current master 7ef8c9144dd38bb59726e82af988242b2407aa0f, 'make
check' with the headless backend now fails for me in:

FAIL: internal-screenshot.weston

test-client: got seat 0x236b6e0 capabilities: 3
test-client: got seat 0x236b6e0 name: 'default'
test-client: got global pointer 100 100
test-client: got seat 0x236bbe0 capabilities: 7
test-client: got seat 0x236bbe0 name: 'test-seat'
test-client: got keyboard repeat_info 40 400
test-client: got keyboard keymap
test-client: got surface enter output 0x236b820
test-client: got keyboard modifiers 0 0 0 0
test-client: got pointer enter 0 0, surface 0x236bd50
internal-screenshot.weston: /home/pq/git/weston/tests/internal-screenshot-test.c:211: internal_screenshot: Assertion `reference' failed.
Creating client for test
Taking a screenshot
Screenshot has been captured
Loading reference image ./tests/reference/internal-screenshot-00.png
Loading reference image ./tests/reference/internal-screenshot-00.png
Could not open ./tests/reference/internal-screenshot-00.png: file not found
test "internal_screenshot":     signal 6, fail.
1 tests, 0 pass, 0 skip, 1 fail


This is an out-of-tree build.

If I run
$ make check TESTS=internal-screenshot.weston WESTON_TEST_REFERENCE_PATH=~/git/weston/tests/reference
the test will pass. This should not be necessary.


Another failure:

$ make check TESTS=internal-screenshot.weston WESTON_TEST_REFERENCE_PATH=~/git/weston/tests/reference BACKEND=x11-backend.so

FAIL: internal-screenshot.weston

test-client: got seat 0x73a6e0 capabilities: 3
test-client: got seat 0x73a6e0 name: 'default'
test-client: got global pointer 100 100
test-client: got seat 0x73abe0 capabilities: 7
test-client: got seat 0x73abe0 name: 'test-seat'
test-client: got keyboard repeat_info 40 400
test-client: got keyboard keymap
test-client: got surface enter output 0x73a820
test-client: got keyboard modifiers 0 0 0 0
test-client: got pointer enter 0 0, surface 0x73ad50
Creating client for test
Taking a screenshot
Screenshot has been captured
Loading reference image /home/pq/git/weston/tests/reference/internal-screenshot-00.png
Loading reference image /home/pq/git/weston/tests/reference/internal-screenshot-00.png
Screenshot different from reference image
Clip: 50,50 101 x 101
Bytewise comparison inside clip
Mismatched image on row 50
50,0:        0       7b  <---
50,1:        0       7b  <---
...
50,374:        0       7b  <---
50,375:        0       7b  internal-screenshot.weston: /home/pq/git/weston/tests/internal-screenshot-test.c:242: inter
nal_screenshot: Assertion `match' failed.
<---
50,376:        0       7b  <---
50,377:        0       7b  <---
...
Screenshot doesn't match reference image in clipped area
Writing PNG to disk
Test complete
test "internal_screenshot":     signal 6, fail.
1 tests, 0 pass, 0 skip, 1 fail


The written screenshot is completely black. I believe this is still due
to the race between weston-desktop-shell and the test client.

I reported and explained both bugs also the last time. I consider at
least the first one (reference image not found) to be a release
blocker. If an out-of-tree build fails 'make check', also 'make
distcheck' will fail, and 'make distcheck' passing is a requirement for
releases.

Note that this is different to the case of
DISTCHECK_CONFIGURE_FLAGS="--with-xserver-path=$WLD/bin/Xwayland"
because Xwayland is an external dependency while the reference images
are internal to the Weston tree.


Thanks,
pq