[RFC PATCH 0/8] Meson build system

Wed Nov 30 21:54:55 UTC 2016

Hi Daniel,

Thanks for doing this!

I share a lot of the same background - I've worked with autotools for many
years, know them well and have also been hesitant to move away. I've
recently ported a small personal project from autotools to meson and I
found it very nice and easy to work with. It feels like a better fit for
the kind of workflow that I'm used to with autotools than many of the other
newer build systems out there.

In terms of speed, it's certainly nice that meson ends up using ninja, but
the much faster configure step is what I'm excited about. Especially on a
multicore system where the configure step runs single-threaded and
bottlenecks the rest of the otherwise parallelizable build process. And the
fact that meson is fast enough that there's no point in a separate
autogen.sh step, make the overall system much simpler and faster.

Of course, one of the points of the generate step in autotools is that you
ship the generated files with your tarballs and the end results only
requires standard sh and make. I don't think meson has a similar feature,
but I'd argue that today requiring python isn't going to limit you any more
than requiring sh and make.

I think having both meson and autotools side by side for a bit would be a
good way to start and I'd be in favor of moving to meson exclusively a few
releases down the road.

Kristian

On Tue, Nov 29, 2016 at 9:00 AM Daniel Stone <daniels at collabora.com> wrote:

> Hi all,
> Whilst working on the atomic series, and the recent review spree, I
> started to really lose patience with how incredibly slow our build
> system is.
>
> This patchset provides a working port to Meson, a Python-based build
> system with a Ninja backend. Whilst autotools runs a combination of
> shell and Perl (aclocal/automake/autoconf) to generate the build files
> and Make and shell to run it, Meson runs its configuration step in
> Python, and outputs Ninja build files.
>
> (tl;dr: smaller platforms 3-6x quicker, bigger platforms 2-4x quicker)
>
> There are a few things which attracted me to Meson. It's really,
> mindblowingly, fast. There's no artificial split between configure and
> build stages (i.e. configure.ac vs. Makefile.am). Its dependencies are
> quite lightweight. It has first-class support for pkg-config, and I've
> not had a problem with executables, shared libraries, modules, static
> helper libraries, binaries to install to libexec, tests[0], etc etc.
>
> I've been keeping an eye on it for a while as it matures; one of the
> things which interested me was seeing GStreamer and GNOME start to
> migrate some of their projects towards it. Not because they're
> incredible and we should blindly follow everything they do, but because
> they have a very similar set of build-system requirements to us, and
> they have also been very conservative in migrating away from autotools
> to date.
>
> I'm incredibly familiar with autotools, dating back to 2004 when I
> started experimenting with converting the X server to use it, which
> later grew into the xserver module we have today. As new build systems
> have come around, I've made a point of looking into them, and trying to
> use them.
>
> SCons and waf were immediately disqualified, because I think writing
> your entire build in actual Python is insane, and too large a barrier to
> entry. CMake is brutally slow[1], I find its interface very unintuitive,
> and cmake --help-full splits out a 37,399-line Markup file to stdout. GN
> (used for Chrome) is very nice, and I prefer its system of 'configs' to
> Meson's per-target flags, but isn't really portable. Most of the rest
> focus on languages we don't use, and/or bring massive dependencies.
>
> Meson's the first one I've found that strikes the right balance (not
> perfect, but within striking distance) of lightweight DSL vs. useful
> programming language. And, it's really incredibly fast. It has far
> better support for cross-compilation and sysroots than autotools can
> ever hope to have. And it's got quite a good upstream community behind
> it, and solid documentation.
>
> I've done a port for Meson, with the full range of options and
> configuration available in autotools today. Currently we're missing
> Doxygen support (because I haven't got around to it) in Wayland, plus I
> haven't been able to test the Weston RDP backend (because it doesn't
> build in Fedora 25) or the fbdev backend (because it's 2016), but they
> are there. There's also currently no support for 'auto' build options; I
> need to look into how best to do that. A couple of the tests in Weston
> are missing, and I think also PAM support for weston-launch.
>
> There are also some inefficiencies: currently, for some reason, Meson
> will run wayland-scanner once for every generated file _for every
> target_. So xdg-shell-unstable-v5-client-protocol.h, for example, gets
> built a bazillion times. Fixing that upstream would actually visibly cut
> the build times. We could also get a speed boost by moving away from
> config.h and using per-target build flags, to avoid unnecessary
> rebuilds: this doesn't work with autotools as it only rebuilds if
> dependent mtimes change, but Meson/Ninja hash the build flags and force
> a rebuild if those change. I'm going to be looking into that later.
>
> It's also smaller than our existing files: 374 lines for Meson versus
> 481 lines for configure.ac/Makefile.am (excluding doc) in Wayland, and
> 1602 lines for Meson versus 2373 lines for
> configure.ac/Makefile.am/weston-tests-env in Weston.
>
> This conversion requires Meson 0.36, which is available in Fedora 25 and
> Debian testing/stretch. It's not available in Ubuntu release versions or
> Debian stable/jessie; it can, however, be easily installed through pip
> on these systems.
>
> Thanks for this go to the very helpful people on Freenode #mesonbuild,
> as well as to the authors of the GStreamer Meson port, who stated that
> they did not assert copyrightability over their build files, allowing
> me to use some of those as inspiration for this port.
>
>
> Anyway, here are some benchmarks I've done on build times, across the
> full range of the computing-power spectrum:
>
> System 1 ('RPi2'): Raspberry Pi 2, 900MHz quad-core ARM Cortex-A7, 1GB
> RAM, storage on SD card, make -j8
>
> System 2 ('Rock2'): Radxa Rock2 Square, Rockchip RK3288 SOC, 1.6GHz
> quad-core ARM Cortex-A17, 2GB RAM, storage on eMMC, make -j8
>
> System 3 ('Laptop'): Dell XPS 13 9350, 2.5GHz dual-core Intel Skylake
> i7-6500U, 4MB cache, 16GB RAM, storage on NVME/PCI-E, make -j8
>
> System 4 ('Xeon'): HP Z840, 2.4GHz 14-core Intel Broadwell Xeon
> E5-2680v4, 35MB cache, 128GB RAM, storage on NVME/PCI-E, make -j18[2]
>
>
> Wayland
> -------
>
> Documentation is disabled to give an apples-to-apples comparison; also
> Meson's default use of ccache is disabled for the same reason. 'make
> install' is split into a separate make invocation, because it fails in a
> dependency-relinking catastrophe otherwise.
>
> Note that the time involving test is mostly dominated by sanity-test,
> which takes around 5.7s (!). If you subtract that from the runtimes, the
> differentials for the faster machines becomes far more apparent.
>
> Test 1 ('full'): full configuration, build, test, and install
>   - time (../autogen.sh --prefix=/tmp/wayland-autotools
> --disable-documentation && make -j$JOBS check && make install)
>   - time (CC=gcc meson --prefix=/tmp/wayland-meson .. && ninja test
> install)
>
> Test 2 ('rebuild'): full rebuild and install
>   - touch ../src/wayland-util.h && time (make -j$JOBS && make check
> install)
>   - touch ../src/wayland-util.h && time (ninja test install)
>
> Test 3 ('check'): relink libweston and test
>   - touch ../src/wayland-server.c && time (make -j$JOBS check)
>   - touch ../src/wayland-server.c && time (ninja test)
>
> Test 4 ('install'): no changes, install
>   - time make install
>   - time ninja install
>
>          |             RPi2 |           Rock2 |          Laptop |
>  Xeon |
>
> ---------+------------------+-----------------+-----------------+----------------+
> Full     | 353.34s / 65.78s | 96.80s / 25.30s | 27.78s / 11.24s | 22.71s /
> 7.19s |
> Rebuild  | 103.73s / 53.83s | 43.61s / 16.17s | 13.80s /  9.63s | 10.87s /
> 6.12s |
> Check    |  44.67s / 14.76s | 17.63s /  7.23s | 13.22s /  8.80s |  6.67s /
> 5.41s |
> Install  |   6.39s /  3.46s |  1.71s /  1.13s |  0.47s /  0.14s |  0.75s /
> 0.36s |
>
> ---------+------------------+-----------------+-----------------+----------------+
>
>
> Weston
> -------
>
> Documentation is disabled to give an apples-to-apples comparison; also
> Meson's default use of ccache is disabled for the same reason. 'make
> install' is split into a separate make invocation, because it fails in a
> dependency-relinking catastrophe otherwise. The RDP backend is disabled
> for both, so I can actually build it on Fedora.
>
> Test 1 ('full'): full configuration, build, test, and install
>   - time (../autogen.sh --prefix=/tmp/weston-autotools
> --disable-rdp-compositor --disable-devdocs --disable-setuid-install && make
> -j$JOBS check && make install)
>   - time (CC=gcc meson --prefix=/tmp/weston-meson -Dbackend_rdp=false ..
> && ninja test install)
>   + -Dclients_dmabuf_drm=false / --disable-simple-dmabuf-intel-client on
> the ARM systems
>
> Test 2 ('rebuild'): full rebuild and install
>   - touch ../libweston/compositor.h && time (make -j$JOBS && make check
> install)
>   - touch ../libweston/compositor.h && time (ninja test install)
>
> Test 3 ('check'): relink libweston and test
>   - touch ../libweston/timeline.c && time (make -j$JOBS check)
>   - touch ../libweston/timeline.c && time (ninja test)
>
> Test 4 ('install'): no changes, install
>   - time make install
>   - time ninja install
>
>          |              RPi2 |            Rock2 |          Laptop |
>    Xeon |
>
> ---------+-------------------+------------------+-----------------+----------------+
> Full     | 685.80s / 163.20s | 188.55s / 34.55s | 53.96s / 12.47s | 25.78s
> / 6.26s |
> Rebuild  | 231.78s /  77.62s |  62.26s / 19.45s | 25.75s /  6.81s | 14.25s
> / 4.80s |
> Check    |  37.21s /  27.18s |  12.00s /  6.82s |  4.91s /  3.22s |  4.42s
> / 3.65s |
> Install  |  21.32s /   7.31s |   7.03s /  1.98s |  1.63s /  0.38s |  2.23s
> / 0.73s |
>
> ---------+-------------------+------------------+-----------------+----------------+
>
>
> Cheers,
> Daniel
>
> [0]: Support for skipping tests has still not quite yet landed, which is
>      a black mark indeed: https://github.com/mesonbuild/meson/pull/1110
> [1]: cf. https://lwn.net/Articles/706653/ and thread
> [2]: This would be -j28, but the build just fails at higher parallelism
>      levels. I don't know why.
>
> _______________________________________________
> wayland-devel mailing list
> wayland-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/wayland-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20161130/cafbe1eb/attachment-0001.html>