[cairo] What does it take to get a make check to pass with the xcb target, using CAIRO_REF_DIR?

Wed Jul 13 14:02:31 UTC 2016

On 07/08, darxus wrote:
> Those tests were not useful, because I ran them against references I generated from the latest commit. It'll be several days before I can try something more useful.

I modified my script to, for each random commit, generate a reference
directory, then run the xcb tests against that new reference directory
(which should always pass).  

I noticed many commits were failing to build with "undefined reference to
`cairo_boilerplate_xmalloc'", apparently caused by
https://bugs.freedesktop.org/show_bug.cgi?id=77060 and fixed by:

commit c7ff9bb32e20679d6da4e8a2856be716e5bd9e12
Author:     Uli Schlachter <psychon at znc.in>
Commit:     Bryce Harrington <bryce at osg.samsung.com>
CommitDate: Fri Aug 1 13:11:31 2014 -0700
    Remove LTO support

That commit mentions -flto was added in:

commit 0870c6fb5b39dcc04fa376123848adde2d06d2ce
Author:     Chris Wilson <chris at chris-wilson.co.uk>
Commit:     Chris Wilson <chris at chris-wilson.co.uk>
CommitDate: Wed May 12 20:54:49 2010 +0100
    gcc-4.5 warnings and optimisation flags.

So I believe from May 2010 to August 2014, commits are failing because of
-flto.  And I suspect commits before then are all failing for other
reasons.

And I believe all commits since then are failing to pass xcb tests.

I am uncertain of this because sometimes Xvfb fails, then all tests return
UNTESTED, and the suite returns PASS.  And then the test-suite.log contains
only the heading saying it passed, and none of the per-test results which
would make it distinguishable from a case where all the tests are actually
passing.  Which is also upsetting.

I'm considering trying stripping out -flto with sed to search further back.

I continue to think it would be a good idea to disable tests which are
currently always failing with a fresh set of references, so that the test
suite is more usable.

Some commit failures (I believe I fixed the timestamps):

$ grep 'FAIL:  1' */2test-suite.log 
1408651002_183ac1889f369743a86a8a2ff6b61486acda614e/2test-suite.log:# FAIL:  1  Thu Aug 21 15:56:42 EDT 2014
1409199266_e6bf829d8953fac82ccc80618c6cae41346e483f/2test-suite.log:# FAIL:  1  Thu Aug 28 00:14:26 EDT 2014
1410697932_0aa43ed886c0f8468a21a470f2f024bd4d8a4513/2test-suite.log:# FAIL:  1  Sun Sep 14 08:32:12 EDT 2014
1411010737_33a54e7c204c4bb7fb0f1f2844608a78ea317db4/2test-suite.log:# FAIL:  1  Wed Sep 17 23:25:37 EDT 2014
1411497603_9a19ef185884da1e7d5344b301f48aa2ba0f88b6/2test-suite.log:# FAIL:  1  Tue Sep 23 14:40:03 EDT 2014
1412081825_167561f2823767058e2be3a26131b5f820b35c35/2test-suite.log:# FAIL:  1  Tue Sep 30 08:57:05 EDT 2014
1412090133_0c42d5c176b27725ac8ab293c3e941be64f51613/2test-suite.log:# FAIL:  1  Tue Sep 30 11:15:33 EDT 2014
1412236004_1a32ce83e4c1f3ca22a3f137b0126003a0e3d6e7/2test-suite.log:# FAIL:  1  Thu Oct  2 03:46:44 EDT 2014
1412236004_79c8b14b9fd8716d47fab4436b34d26b0135e9d7/2test-suite.log:# FAIL:  1  Thu Oct  2 03:46:44 EDT 2014
1412904533_82cd66f8330da0521c29c003ef0a4bb773d6dbc6/2test-suite.log:# FAIL:  1  Thu Oct  9 21:28:53 EDT 2014
1413110205_ab2a7b61d59258b71e1c3256edf2f42ea6cc1eb2/2test-suite.log:# FAIL:  1  Sun Oct 12 06:36:45 EDT 2014
1420293591_02e4efc961be40d266d4df0acaf3271219529017/2test-suite.log:# FAIL:  1  Sat Jan  3 08:59:51 EST 2015
1425698353_246d5c1d477cc246788f5695a08c4c414a36fbf5/2test-suite.log:# FAIL:  1  Fri Mar  6 22:19:13 EST 2015
1426718981_904f1fdc1d7f13d4443bafd760db7395c5030ca7/2test-suite.log:# FAIL:  1  Wed Mar 18 18:49:41 EDT 2015
1428187758_2cf2d8e340a325adb205baf8e4bd64e1d1858008/2test-suite.log:# FAIL:  1  Sat Apr  4 18:49:18 EDT 2015
1428972208_16898ba11b4d6e9e2e64bb2d02d0fb5adbe266e2/2test-suite.log:# FAIL:  1  Mon Apr 13 20:43:28 EDT 2015
1429176530_89942c6af00af8b72babeae676376f4cf8911c16/2test-suite.log:# FAIL:  1  Thu Apr 16 05:28:50 EDT 2015
1445069359_30eb1b0e9778a96c2eb984f243dec5e3c8859e05/2test-suite.log:# FAIL:  1  Sat Oct 17 04:09:19 EDT 2015
1449823502_db8a7f1697c49ae4942d2aa49eed52dd73dd9c7a/2test-suite.log:# FAIL:  1  Fri Dec 11 03:45:02 EST 2015
1465125216_1e07ced66d26475e6631df9ffa2a15709104bd8f/2test-suite.log:# FAIL:  1  Sun Jun  5 07:13:36 EDT 2016
1467845160_master/2test-suite.log:# FAIL:  1                                    Wed Jul  6 18:46:00 EDT 2016

Dates were printed with: 
$ grep 'FAIL:  1' */2test-suite.log  | cut -d'_' -f1 | xargs -Iunixtime date -d'@unixtime'

Failure rate by test name:
$ grep -h ': FAIL' */2test-suite.log  | cut -d':' -f1 | sort | uniq -c | sort -nr
     13 record2x-paint-alpha-clip-mask
     12 recordflip-paint-alpha-clip-mask
     11 record90-paint-alpha-clip-mask
     11 record1414x-paint-alpha-clip-mask
     10 record-paint-alpha-clip-mask
      9 create-from-png

More detail:

$ grep -h ^record-paint-alpha-clip-mask */2test-suite.log
record-paint-alpha-clip-mask: UNTESTED
record-paint-alpha-clip-mask: UNTESTED
record-paint-alpha-clip-mask: UNTESTED
record-paint-alpha-clip-mask: UNTESTED
record-paint-alpha-clip-mask: UNTESTED
record-paint-alpha-clip-mask: UNTESTED
record-paint-alpha-clip-mask: PASS
record-paint-alpha-clip-mask: FAIL (xcb, xcb-window&)
record-paint-alpha-clip-mask: UNTESTED
record-paint-alpha-clip-mask: FAIL (xcb, xcb-window&)
record-paint-alpha-clip-mask: PASS
record-paint-alpha-clip-mask: FAIL (xcb, xcb-window, xcb-window&)
record-paint-alpha-clip-mask: FAIL (xcb, xcb-window, xcb-window&)
record-paint-alpha-clip-mask: FAIL (xcb, xcb-window)
record-paint-alpha-clip-mask: FAIL (xcb, xcb-window, xcb-window&)
record-paint-alpha-clip-mask: FAIL (xcb, xcb-window)
record-paint-alpha-clip-mask: FAIL (xcb)
record-paint-alpha-clip-mask: FAIL (xcb, xcb-window&)
record-paint-alpha-clip-mask: PASS
record-paint-alpha-clip-mask: FAIL (xcb, xcb-window, xcb-window&)
record-paint-alpha-clip-mask: PASS

$ grep -h ^record2x-paint-alpha-clip-mask */2test-suite.log
record2x-paint-alpha-clip-mask: UNTESTED
record2x-paint-alpha-clip-mask: UNTESTED
record2x-paint-alpha-clip-mask: UNTESTED
record2x-paint-alpha-clip-mask: UNTESTED
record2x-paint-alpha-clip-mask: UNTESTED
record2x-paint-alpha-clip-mask: UNTESTED
record2x-paint-alpha-clip-mask: PASS
record2x-paint-alpha-clip-mask: FAIL (xcb, xcb-window, xcb-window&)
record2x-paint-alpha-clip-mask: UNTESTED
record2x-paint-alpha-clip-mask: FAIL (xcb, xcb-window&)
record2x-paint-alpha-clip-mask: FAIL (xcb, xcb-window, xcb-window&)
record2x-paint-alpha-clip-mask: FAIL (xcb, xcb-window, xcb-window&)
record2x-paint-alpha-clip-mask: FAIL (xcb, xcb-window, xcb-window&)
record2x-paint-alpha-clip-mask: FAIL (xcb, xcb-window, xcb-window&)
record2x-paint-alpha-clip-mask: FAIL (xcb-window)
record2x-paint-alpha-clip-mask: FAIL (xcb, xcb-window, xcb-window&)
record2x-paint-alpha-clip-mask: FAIL (xcb, xcb-window)
record2x-paint-alpha-clip-mask: FAIL (xcb, xcb-window)
record2x-paint-alpha-clip-mask: FAIL (xcb, xcb-window, xcb-window&)
record2x-paint-alpha-clip-mask: FAIL (xcb, xcb-window&)
record2x-paint-alpha-clip-mask: FAIL (xcb, xcb-window, xcb-window&)

Hmm, for
1412090133_0c42d5c176b27725ac8ab293c3e941be64f51613 all record* tests
passed, create-from-png failed.  