GNU make with depcache/some general build speed up tips/three LibreOffice GNU make forks considered harmful

[Bjoern Michaelsen]

Hi,

as promised at FOSDEM here is the proof-of-concept for the dependency caching
stuff for GNU make. To get it do:

 git clone https://github.com/bjoernmichaelsen/gnu-make-depcache.git gnu-make
 git checkout feature/depcache
 git clean -dfx
 autoreconf -i && ./configure && make update && make && make check

and then point the "export GNUMAKE=" line in your config_host.mk to the
generated binary. Here is what is there:
- cache generation
- cache usage
- basic test harnish
- backwards compatible implementation (triggering on .FEATURES variable)

Here is what is missing:
- handling missing dependency files (gotta look into that one)
- documentation
- the depcache isnt platform-independant

Below are the numbers -- note that the 32 seconds are on the same machine as
the numbers I presented at FOSDEM, but instead of doing measurements on the
bus, I did so in a calm place at home.

Conclusions so far:
- measuring IO performance on a notebook is tricky, doing so on the bus to
  FOSDEM is insane
- unplugging your power supply can ~half your IO speed on a SSD/notebook even
  on Linux
- Maintainer Mode is slow, it does a lot of extra sanity checks at runtime
  if you use a self compiled GNU make you really want:
  https://github.com/bjoernmichaelsen/gnu-make-depcache/commit/00bc33b760cc54a270418212af2d4750808b91f9
- tmpfs has no additional advantage when you do depcaching
- we are below 11 seconds for build-nocheck on a 3 year old dev notebook, no
  need to push this further

I would really love to hear, if the depcache does have any impact at all on:
- slow HDDs (not ssds)
- other platforms (esp. slow Windows)

To test this, do a full build, get the depcache branch and build it as
described above, and then edit the "export GNUMAKE" line in your config_host.mk
to point to the modified make executable.

For the Linux build, its not worth fixing the remaining issues -- but if
Windows gains from this, I would have a look.

Best,

Bjoern

P.S.: We current have two gnu make forks in dev-tools, one in the gnu-make-lo
      repository on gerrit, were one cant push to. Such a mess. When nobody
      protests, I will delete the copies in dev-tools and make the gnu-make-lo
      repository writable. Whatever forking had been done there and is still of
      value needs to be pushed on proper branches there then.

build times:

distro provided GNU make 3.81, power supply unplugged, build on SSD:
real    0m32.942s

distro provided GNU make 3.81, power supply plugged, build on SSD:
real    0m17.071s

GNU make 4.0 -- maintainer mode, power supply plugged, build on SSD:
real    0m18.469s

GNU make 4.0 -- no maintainer mode, power supply plugged, build on SSD:
real    0m16.607s

4.0, no maintainer mode, irst run, caching dependencies, power supply plugged, build on SSD:
real    0m22.530s

4.0, no maintainer mode, econd run, using cache, power supply plugged, build on SSD:
real    0m11.070s

4.0, no maintainer mode, second run, using cache, power supply plugged, build on tmpfs:
real    0m11.155s