Release blocker (Re: [PATCH weston v3 00/15] Implement screenshot-based testing with the headless renderer)

[Bryce Harrington]

On Fri, May 22, 2015 at 10:34:00AM +0300, Pekka Paalanen wrote:
> 
> Another failure:
> 
> $ make check TESTS=internal-screenshot.weston WESTON_TEST_REFERENCE_PATH=~/git/weston/tests/reference BACKEND=x11-backend.so
> 
> FAIL: internal-screenshot.weston
> test-client: got seat 0x73a6e0 capabilities: 3
> test-client: got seat 0x73a6e0 name: 'default'
> test-client: got global pointer 100 100
> test-client: got seat 0x73abe0 capabilities: 7
> test-client: got seat 0x73abe0 name: 'test-seat'
> test-client: got keyboard repeat_info 40 400
> test-client: got keyboard keymap
> test-client: got surface enter output 0x73a820
> test-client: got keyboard modifiers 0 0 0 0
> test-client: got pointer enter 0 0, surface 0x73ad50
> Creating client for test
> Taking a screenshot
> Screenshot has been captured
> Loading reference image /home/pq/git/weston/tests/reference/internal-screenshot-00.png
> Loading reference image /home/pq/git/weston/tests/reference/internal-screenshot-00.png
> Screenshot different from reference image
> Clip: 50,50 101 x 101
> Bytewise comparison inside clip
> Mismatched image on row 50
> 50,0:        0       7b  <---
> 50,1:        0       7b  <---
> ...
> 50,374:        0       7b  <---
> 50,375:        0       7b  internal-screenshot.weston: /home/pq/git/weston/tests/internal-screenshot-test.c:242: inter
> nal_screenshot: Assertion `match' failed.
> <---
> 50,376:        0       7b  <---
> 50,377:        0       7b  <---
> ...
> Screenshot doesn't match reference image in clipped area
> Writing PNG to disk
> Test complete
> test "internal_screenshot":     signal 6, fail.
> 1 tests, 0 pass, 0 skip, 1 fail
> 
> 
> The written screenshot is completely black. I believe this is still due
> to the race between weston-desktop-shell and the test client.

This afternoon I've tried to reproduce this behavior, but it seems to be
working for me.  Derek also is unable to reproduce.

I did discover that if I ssh -X in to the box and run make check over
ssh using the X11 backend, then I get a test failure.  But that's not
due to the test; same failure occurs over ssh if I run

 ../weston-build/weston --use-pixman --backend=x11-backend.so

The serverlog shows this causes an xcb "error 10" whatever that means.
But this is completely unrelated to your problem (and probably
irrelevant to anyone but me).

Anyway, I'll hold off on doing the release until you're happy.

Bryce
 
> I reported and explained both bugs also the last time. I consider at
> least the first one (reference image not found) to be a release
> blocker. If an out-of-tree build fails 'make check', also 'make
> distcheck' will fail, and 'make distcheck' passing is a requirement for
> releases.
> 
> Note that this is different to the case of
> DISTCHECK_CONFIGURE_FLAGS="--with-xserver-path=$WLD/bin/Xwayland"
> because Xwayland is an external dependency while the reference images
> are internal to the Weston tree.
> 
> 
> Thanks,
> pq