My Windows build is much slower than Linux build. Any way to speed up?

[bjoern]

Hi,

On Mon, Dec 23, 2013 at 02:30:33PM +0100, Christian Lohmaier wrote:
> Just compare
> 
> mkdir foo ; cd foo
> time touch {1..1000}
> 
> with
> 
> mkdir foo; cd foo
> time for i in {1..1000}; do touch $i ; done
> 
> While on linux there's no real difference, on windows this is more
> than an order of magnitude...(and the build system does the equivalent
> of the latter a lot)

Yes, AFAIK currently there are two main contributions to the build on windows
being slower:
- the MSVC compiler being slower itself
- process creation on windows (or cygwin or shell in cygwin) being slow (see
  Clophs example about)

There isnt so much we can do about the first thing in a clean way. We can do
something about the second thing, e.g.:

 http://cgit.freedesktop.org/libreoffice/core/commit/?id=66a0713dc9c676182fcd7aa1e21f8dc25c05be5e

improved Windows build time by 10%. Some other things to consider doing for
Windows optimization:

- IIRC MSVC can compile multiple object files with one call.
  Obviously all the compile flags etc. must be the same then for all objects to
  create, but for cxx files from the same dir and for the same library this is
  given. So one could e.g. always batch up 8 of such objects and compile them
  with one call. But I fear this is adding a huge load of complexity to the build, because:
  - it IIRC does not generate per object dependency files then
  - in incremental builds, you would only need to rebuild _some_ files -- so
    you cant do the grouping of objects in some simple or static way (as you
    would 'overrebuild' then)
  - it also collides with parallelization in the build sytem -- if you put e.g.
    always 8 object in one call and only have to rebuild 8 objects (a common
    developer change), you would get no parallelization at all

- as noted in the comment above the touching of a file in cygwin took some
  ~250ms -- interestingly, this is not CPU-bound. A simplistic approach thus
  would be, to just allways overcommit the build by some factor and see if it
  improves the build time (or the CPU load).

Of course, the real issue is: what is this 250ms idle wait to touch a file from
cygwin about? If we know that we can likely workaround it. Maybe its cygwin
polling some network share for the cygwin->windows path mapping? So any
detective work on that might help:

- Can one make the delay go away by e.g. disabling network shares/virus
  scanners/other services?
- Does this delay also appear when doing network-IO instead of file-IO
- If so, could we e.g. start a (native-Windows) Python demon that listens on a
  network socket and hand it the commands to execute as be faster?
- Or, does it make sense to make the gbuild in cygwin dump e.g. output for the
  ninja (the minimalistic build backend) and then have a (native windows) ninja
  process execute this.

Finally, Michael Stahl is considering moving gbuild to work on a native windows
GNU make without the cygwin wrapper. But that likely is quite a bit of work.

So, you see there is a good bit to research (where does the 250 ms delay on
cygwin come from), to evaluate (batch builds, handing over to a native demon,
creating output for ninja, using native GNU make). TBH though, I fear most of
the hacks above -- with the exception of running native GNU make -- make the
build system even more complex and harder to maintain, so I am not convinced
that its an good idea to push for them without very well-funded research
covering all cases.

Best,

Bjoern