[cairo] What does it take to get a make check to pass with the xcb target, using CAIRO_REF_DIR?

Fri Jul 8 21:55:12 UTC 2016

Those tests were not useful, because I ran them against references I generated from the latest commit. It'll be several days before I can try something more useful.

-------- Original message --------
From: darxus at chaosreigns.com 
Date: 07/08/2016  8:58 AM  (GMT-05:00) 
To: Bryce Harrington <bryce at osg.samsung.com> 
Cc: cairo at cairographics.org 
Subject: Re: [cairo] What does it take to get a make check to pass with the
  xcb target, using CAIRO_REF_DIR? 

On 07/07, Bryce Harrington wrote:
> When doing releases I also try to do a run and compare with the prior
> release to spot regressions.  I'd really love to be able to do a
> thorough git bisect search to locate which changes caused which tests to
> fail, but as the suite takes a while to run it'd be pretty time
> consuming.

I think bisecting sounds like great fun, except for the irregularity of
problems.  I wrote a script to build a random commit, back through 2010
(because they seemed to not be successfully building before that).  All of
them either gave me no test-suite.log, or, for the 22 that gave me a log,
the test suite had at least 6 failures.  

Nothing passed all tests.  Does anybody have any ideas when all tests for
the xcb target might have passed?  Is this evidence that the FAILs are
caused by an external library?

This is the output, plus the perl script I wrote:
http://www.chaosreigns.com/cairo/tests.tar.gz
(The script is kind of crude.)

This is a filename with a unix timestamp (git log -1 --pretty=format:%ct),
an underscore, the commit hash, a colon, the number of FAILs, some spaces,
then a GMT date calculated from the unix timestamp:

darxus at dancer:~/source/cairolog$ grep -l test * | xargs grep -c ' FAIL'
1264201313_c0458b456007f718747be7fd690e674df5026059:7    Fri Jan 22 23:01:53 2010
1269204078_247ef2bd63e755963f7930ac06c79b95aed2adb4:6    Sun Mar 21 20:41:18 2010
1271277994_54670ec13d64efa94f552b5473c1f15a9db1cecd:145  Wed Apr 14 20:46:34 2010
1273693338_094f0e0fa0153f290061635eed51e8d1dbe2cf4a:5    Wed May 12 19:42:18 2010
1276288178_d31e2418d6d9920205a154f21becca12a7b84c2a:7    Fri Jun 11 20:29:38 2010
1278883932_fbb0a260b707cb5f02a14cc368c6f2f0d63564c3:146  Sun Jul 11 21:32:12 2010
1286969384_90d50cd92315d6760069ad8062aba5e297370b20:6    Wed Oct 13 11:29:44 2010
1292229969_d9ba8337ab456ae0e232d3c603cb41cea984ebea:32   Mon Dec 13 08:46:09 2010
1292427140_e4b78424ac82588bcb9b855d5b6d5872050d33f9:7    Wed Dec 15 15:32:20 2010
1297614844_c3605bd3adcae1f12731230a1ea599d15e3c8cad:7    Sun Feb 13 16:34:04 2011
1311937629_58df191946b7d275c5706dc40e1cf3dffa7cceae:8    Fri Jul 29 11:07:09 2011
1313177159_bb24f165ff57973347b34956a371c6b33d2d9b59:55   Fri Aug 12 19:25:59 2011
1342113299_3ed581149f82b653e1987e746467ad8ae9ce3ca2:7    Thu Jul 12 17:14:59 2012
1348565464_31291611625c21bc9ec2827cf4ccc8a1f18392ed:55   Tue Sep 25 09:31:04 2012
1405128883_d5ffe67008afe46c030b31afc0803321b6479990:7    Sat Jul 12 01:34:43 2014
1413893554_f032133e6d5ad05157fc46609d8c63103028342c:7    Tue Oct 21 12:12:34 2014
1423145626_3cf862f6d973755cd9824c2224fbe0a623c47ff1:7    Thu Feb  5 14:13:46 2015
1425713393_58df191946b7d275c5706dc40e1cf3dffa7cceae:7    Sat Mar  7 07:29:53 2015
1434656331_ae608035c7b7133826a608d45e067c3875a1aceb:7    Thu Jun 18 19:38:51 2015
1435360353_70cc8f250b5669e757b4f044571ba0f71e3dea9e:7    Fri Jun 26 23:12:33 2015
1465125216_caa4c9fdeb3aecd9a4288114e75d24ec931cd01b:6    Sun Jun  5 11:13:36 2016
1467845160_f9b65ae1fc91bc558a01c2ad7be5a121c6f10818:7    Wed Jul  6 22:46:00 2016

Of the ones that had fewer than 10 FAILs, the test names mostly started
with "record", with a few "subsurface".  

The most recent one with no test-suite.log was      
1466331752_bf5adaf3942388e58ad3bda30173e53b214df885 Sun Jun 19 10:22:32 2016 
Which I manually checked, and it did fail to build:
test/cairo-test-runner.c:130: undefined reference to `cairo_boilerplate_xmalloc'

The second most recent one with no test-suite.log was:
1465125216_afe6f4f0519606c4bc7e9b705b0cae75692d7af2 Sun Jun  5 11:13:36 2016 
It also failed to build:
./autogen.sh: running `./configure --cache-file=config.cache --disable-static --enable-test-surfaces --prefix=/home/darxus/install --enable-xcb'
configure: creating cache config.cache
configure: error: cannot find install-sh, install.sh, or shtool in "." "./.." "./../.."

(Which, so far, confirms my expectation that empty output files are due to
failures to build.)

This is how many I attempted to build per year that produced no
test-suite.log.  The ones from before 2010 were from before I started to
restrict the date range (and go to sleep):

$ grep -L test * | cut -d'_' -f1 | ~/bin/unixtime.pl | awk '{print $5}' | uniq -c
      1 2002
      3 2003
      3 2004
     16 2005
     14 2006
     15 2007
     23 2008
      6 2009
    100 2010
     80 2011
     40 2012
     22 2013
     18 2014
      7 2015
      2 2016

This is how many total I ran:

darxus at dancer:~/source/cairolog$ ls *_* | cut -d'_' -f1 | ~/bin/unixtime.pl | awk '{print $5}' | uniq -c
      1 2002
      3 2003
      3 2004
     17 2005
     14 2006
     15 2007
     23 2008
      6 2009
    109 2010
     84 2011
     42 2012
     22 2013
     20 2014
     11 2015
      4 2016

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cairographics.org/archives/cairo/attachments/20160708/5ea244a9/attachment-0001.html>