Using ccache and jom to speed up Windows builds

[Hossein Nourikhah]

Hello,

With the effort from Luboš Luňák in 2021, building LibreOffice master 
with ccache/VS on Windows became possible. But, as discussed in 2021, 
and also a few days ago, the conclusion was that it is not helpful. The 
recommendation in the TDF Wiki is also not to use ccache:

2021 discussion:
FYI: ccache for MSVC/Windows
https://lists.freedesktop.org/archives/libreoffice/2021-November/088061.html

Building LibreOffice on Windows with Cygwin and MSVC: Tips and Tricks
https://wiki.documentfoundation.org/Development/BuildingOnWindows

Last week, I tried to build again, and this time not using binaries of 
the patched ccache for Cygwin, but with the official Windows binaries. 
The binary of the patched Cygwin ccache was no longer working with the 
latest Cygwin, and I was expecting that the latest binaries should 
contain the patches from Luboš.
In addition, I expected that the Windows binary built with MSVC can 
possibly perform better, because it is a single binary without reliance 
on Cygwin, and without its overhead.

I downloaded ccache from the below link, and put the ccache.exe in 
Cygwin binary folder:

cccache 4.8.1 (4.7.5 also work)
Windows x86_64 binary release
https://github.com/ccache/ccache/releases

I also downloaded jom from the below link, and also put it in the Cygwin 
binary folder. As you may know, it is a parallel drop-in replacement for 
nmake:

jom 1.1.3
https://download.qt.io/official_releases/jom/jom_1_1_3.zip

A small change (adding missing double quotes) is needed to compile 
OpenSSL, as it gives error with ccache. This change does not build the 
rest of LibreOffice, so a fix is needed. But, the quick and dirty change 
works to show the performance difference:

diff --git a/external/openssl/ExternalProject_openssl.mk 
b/external/openssl/ExternalProject_openssl.mk
index e44ccf5f3436..dcb49c8d2dc4 100644
--- a/external/openssl/ExternalProject_openssl.mk
+++ b/external/openssl/ExternalProject_openssl.mk
@@ -65,7 +65,7 @@ $(call 
gb_ExternalProject_get_state_target,openssl,build):
         $(call gb_ExternalProject_run,build,\
                 CONFIGURE_INSIST=1 $(PERL) Configure $(OPENSSL_PLATFORM) 
no-tests no-multilib \
                 && export PERL="$(shell cygpath -w $(PERL))" \
-               && nmake -f makefile \
+               && jom -f makefile \
                         $(if $(call 
gb_Module__symbols_enabled,openssl),DEBUG_FLAGS_VALUE="$(gb_DEBUGINFO_FLAGS)") 
\
         )
         $(call gb_Trace_EndRange,openssl,EXTERNAL)
diff --git a/solenv/gbuild/platform/com_MSC_class.mk 
b/solenv/gbuild/platform/com_MSC_class.mk
index ba42b0ba48df..69f5b00b56c5 100644
--- a/solenv/gbuild/platform/com_MSC_class.mk
+++ b/solenv/gbuild/platform/com_MSC_class.mk
@@ -665,7 +665,7 @@ gb_ExternalProject_INCLUDE := \
         $(subst -I,,$(subst $(WHITESPACE),;,$(SOLARINC)))

  gb_NMAKE_VARS = \
-       CC="$(shell cygpath -w $(filter-out -%,$(CC))) $(filter 
-%,$(CC))" \
+       CC="ccache\" \"$(shell cygpath -w $(lastword $(filter-out 
-%,$(CC))))" \
         INCLUDE="$(gb_ExternalProject_INCLUDE)" \
         LIB="$(ILIB)" \
         MAKEFLAGS= \

I build with this configuration:

--enable-ccache
--enable-odk
--enable-dbgutil
--without-doxygen
--with-visual-studio=2022
--without-java

Without ccache, my build finishes in 30 minutes:

$ make
...
real    30m8.900s
user    0m0.109s
sys     0m0.078s

To give an overview on build performance difference, I've compared the 
time needed to build OpenSSL:

openssl, jom, ccache (warm)
real    0m27.745s
user    0m0.000s
sys     0m0.000s

openssl, nmake, ccache (warm)
real    1m33.791s
user    0m0.000s
sys     0m0.015s

openssl: jom, no ccache
real    2m1.111s
user    0m0.000s
sys     0m0.000s

openssl: nmake, no ccache
real    8m4.923s
user    0m0.000s
sys     0m0.015s

Without ccache, the rest of build (excluding OpenSSL) takes ~25 minutes, 
and this will lead to the ~30 minutes time needed to do a full re-build 
after 'make clean'.

real    24m40.578s
user    0m0.156s
sys     0m0.109s

On the other hand, full rebuild (make) with jom for OpenSSL and ccache 
(warm) finishes in 18 minutes, and that is a considerable improvement!

real    17m57.119s
user    0m0.015s
sys     0m0.031s

Please note that with ccache, the build sleeps for a while while showing 
sd, and several MSBuild.exe processes active in the background. The 
build finishes anyway, and the built LibreOffice is usable. Fixing the 
problem there may save a few more minutes:

[build MOD] sc

There may be possible drawbacks. As OpenSSL in not always built and 
tested with jom instead of nmake, there is a chance that it may fail 
with some future changes. But anyway, it is a a good optimization.

I think it worth trying ccache + jom on Windows CI. Also, using jom in 
every external library that is built with nmake can provide more 
performance improvement. One can take a look into:

$ git grep nmake "external/*/*.mk"

Regards,
Hossein

-- 
Hossein Nourikhah, Ph.D., Developer Community Architect
Tel: +49 30 5557992-65 | Email: hossein at libreoffice.org
The Document Foundation, Winterfeldtstraße 52, 10781 Berlin, DE
Gemeinnützige rechtsfähige Stiftung des bürgerlichen Rechts
Legal details: https://www.documentfoundation.org/imprint