minutes of ESC call ...

[Bjoern Michaelsen]

Hi,

On Thu, May 28, 2015 at 06:01:33PM +0000, Noel Grandin wrote:
> Wasn't someone going to look at speeding that up? Make check is not
> currently making very good use of multiple cores and it has a long tail
> where only one or two cores are busy.

I cant fully confirm that -- at least not combined with Michaels 15 minutes
number.  Good ol' big Bertha with two Opteron 6272s has 32 cores, but mediocre
single-threaded performance compared with some of todays offerings. Still[1]:

-j
real     5m 9.091s
user    25m12.250s
sys      2m53.420s

-j32
real     5m30.133s
user    29m46.310s
sys      3m10.660s

-j8
real     6m35.434s
user    19m36.700s
sys      2m 6.390s

-j1
real    34m11.814s
user    17m51.990s
sys      1m52.470s

(all above along with moderate desktop use alongside: Firefox, mplayer audio
streaming etc. -- so not too scientific.)

And with this I see a lot of processes running parallel till the end. This
suggests to me that the stuff is quite parallelized -- however none of the
testing threads seem to be CPU-bound rather the Java-stuff seems to be IO bound
(for IPC?) with the slowest test unit taking ~5 minutes. Thus even 32 jobs or
more dont seem to keep more than one core busy.

At least for people with a meager duo-core laptop or somesuch, this suggests
you should try:

 time make check PARALLELISM=9999 # aka forkbomb me please

And if that doesnt explode your RAM, it should help you quite a bit and you
should end up with 5 minutes for 'make check' -- quite independant of your
hardware, as much of that seems IO-bound idling. If we want to cut down on that
5 minutes, we would need to touch the JunitTest internals.

Best,

Bjoern


[1] As a sidenote: I noted that linking with debug symbols takes at least 4
times as long as without debug symbols. Given that linking happens at the end
of the build with a narrow dependency tree left (as libraries are daisy-chained
dependency-wise for linking), you are really penalized for building with
symbols when you dont need those, as it can easily dry out paralellism. So dont
do that: Rather than building with symbols everywhere and then skipping on
tests, better build without symbols unless you are debugging, but _run the
tests_.

[2] On my ancient notebook with a i7-2720QM (a modern 4th gen i7-4770HQ is at
least 50% faster at the same speed) I find the following with disabling
Junittest_toolkit_unoapi as it was failing on that checkout.

time make check PARALLELISM=9999:
reall    5m15.351s
user    15m52.808s
sys      1m10.073s

time make check (thus -j8):
real      6m2.806s
user    14m32.157s
sys       1m1.861s

time make check PARALLELISM=1:
real    41m 7.984s
user    27m33.174s
sys      1m13.949s