Release blocker (Re: [PATCH weston v3 00/15] Implement screenshot-based testing with the headless renderer)

[Derek Foreman]

On 22/05/15 02:34 AM, Pekka Paalanen wrote:
> On Thu, 21 May 2015 15:43:18 -0700
> Bryce Harrington <bryce at osg.samsung.com> wrote:
> 
>> On Thu, May 21, 2015 at 04:44:41PM -0500, Derek Foreman wrote:
>>> With the tiny stride change mentioned on IRC, which, for those that
>>> weren't present, was basically little more than:
>>> if (stride != bpp * reference->width) FAIL
>>> in load_surface_from_png()
>>>
>>> for the series:
>>> Reviewed-By: Derek Foreman <derekf at osg.samsung.com>
>>
>> Thanks, landed (including the additional stride change):
>>    f556ebe..0ccf8e2  master -> master
> 
> Hi,
> 
> with the current master 7ef8c9144dd38bb59726e82af988242b2407aa0f, 'make
> check' with the headless backend now fails for me in:
> 
> FAIL: internal-screenshot.weston
> 
> test-client: got seat 0x236b6e0 capabilities: 3
> test-client: got seat 0x236b6e0 name: 'default'
> test-client: got global pointer 100 100
> test-client: got seat 0x236bbe0 capabilities: 7
> test-client: got seat 0x236bbe0 name: 'test-seat'
> test-client: got keyboard repeat_info 40 400
> test-client: got keyboard keymap
> test-client: got surface enter output 0x236b820
> test-client: got keyboard modifiers 0 0 0 0
> test-client: got pointer enter 0 0, surface 0x236bd50
> internal-screenshot.weston: /home/pq/git/weston/tests/internal-screenshot-test.c:211: internal_screenshot: Assertion `reference' failed.
> Creating client for test
> Taking a screenshot
> Screenshot has been captured
> Loading reference image ./tests/reference/internal-screenshot-00.png
> Loading reference image ./tests/reference/internal-screenshot-00.png
> Could not open ./tests/reference/internal-screenshot-00.png: file not found
> test "internal_screenshot":     signal 6, fail.
> 1 tests, 0 pass, 0 skip, 1 fail
> 
> 
> This is an out-of-tree build.
> 
> If I run
> $ make check TESTS=internal-screenshot.weston WESTON_TEST_REFERENCE_PATH=~/git/weston/tests/reference
> the test will pass. This should not be necessary.

I think I've just posted a fix for this on list.  It passes for me here
in both out of tree and in tree builds...

> 
> Another failure:
> 
> $ make check TESTS=internal-screenshot.weston WESTON_TEST_REFERENCE_PATH=~/git/weston/tests/reference BACKEND=x11-backend.so
> 
> FAIL: internal-screenshot.weston
> 
> test-client: got seat 0x73a6e0 capabilities: 3
> test-client: got seat 0x73a6e0 name: 'default'
> test-client: got global pointer 100 100
> test-client: got seat 0x73abe0 capabilities: 7
> test-client: got seat 0x73abe0 name: 'test-seat'
> test-client: got keyboard repeat_info 40 400
> test-client: got keyboard keymap
> test-client: got surface enter output 0x73a820
> test-client: got keyboard modifiers 0 0 0 0
> test-client: got pointer enter 0 0, surface 0x73ad50
> Creating client for test
> Taking a screenshot
> Screenshot has been captured
> Loading reference image /home/pq/git/weston/tests/reference/internal-screenshot-00.png
> Loading reference image /home/pq/git/weston/tests/reference/internal-screenshot-00.png
> Screenshot different from reference image
> Clip: 50,50 101 x 101
> Bytewise comparison inside clip
> Mismatched image on row 50
> 50,0:        0       7b  <---
> 50,1:        0       7b  <---
> ...
> 50,374:        0       7b  <---
> 50,375:        0       7b  internal-screenshot.weston: /home/pq/git/weston/tests/internal-screenshot-test.c:242: inter
> nal_screenshot: Assertion `match' failed.
> <---
> 50,376:        0       7b  <---
> 50,377:        0       7b  <---
> ...
> Screenshot doesn't match reference image in clipped area
> Writing PNG to disk
> Test complete
> test "internal_screenshot":     signal 6, fail.
> 1 tests, 0 pass, 0 skip, 1 fail
> 
> 
> The written screenshot is completely black. I believe this is still due
> to the race between weston-desktop-shell and the test client.

I'm going to try to fix this as soon as I can reproduce it.  So far it's
passed every time I've run it...

> I reported and explained both bugs also the last time. I consider at
> least the first one (reference image not found) to be a release
> blocker. If an out-of-tree build fails 'make check', also 'make
> distcheck' will fail, and 'make distcheck' passing is a requirement for
> releases.
> 
> Note that this is different to the case of
> DISTCHECK_CONFIGURE_FLAGS="--with-xserver-path=$WLD/bin/Xwayland"
> because Xwayland is an external dependency while the reference images
> are internal to the Weston tree.
> 
> 
> Thanks,
> pq
>