Experimental build using the Meson build system

Fri Sep 2 18:00:17 UTC 2016

Hi,

We have recently started adding build description files for the Meson
build system to various modules in git master, and I'd like to quickly
explain some of the thinking behind this, where we're at and where
we're planning to go with this.

== How does this affect me as a user, developer, or packager?

In short: it doesn't. The autotools-based build is still our primary
build method and this will not change in the immediate future,
certainly not in the current release cycle (i.e. 1.9/1.10).

However, if you like to tinker, experiment and risk a peek at what
might be our future build system at some point, then this mail is for
you.

== Alright, sign me up! So what's this Meson thing?

Meson is a build system. It features a fairly simple, clean, python-
esque but non-Turing complete build description language. It combines
the functionality of autoconf (configure.ac) and automake (Makefile.am)
into one and can create build files for multiple backends.

The default backend is 'ninja' (https://ninja-build.org) which is a
build tool used by google which is similar in functionality to 'make'.
There are also backends for Microsoft Visual Studio and Apple's Xcode,
although please note that the VS backend and especially the Xcode
backend are not on par with the Ninja backend at this point.

Meson is multi-platform and supports all of the major platforms and
compilers we're targeting, as well as cross-compilation.

== Feature-wise this sounds pretty similar to CMake? Why not just use
CMake? 

There are many reasons for that, both technical and non-technical.
Unlike Meson, CMake has been around for a while and has proven itself
in the field with simple and complex projects. Like Meson, CMake offers
multiple backends and its backends are likely more mature than Meson's.
There are quite a few people who are familiar with it already, which is
always useful.

However, CMake's build description language is just as awful as
autotools' and it has extremely poor support for central components of
our build ecosystem such as pkg-config (intentionally, it seems). There
are other shortcomings as well [1]. It is not the intention of this
post to discuss in detail the pros and cons of CMake or other build
systems, just to mention that it has been considered but was ruled out
for technical reasons as well as aesthetic and what some might call
'cultural' reasons.

It's clear that CMake works great for many people, but it simply has to
be acknowledged that it's not for us.

Having said that, it should be pointed out that Meson took inspiration
from CMake (as well as other build systems) and stole many of the good
ideas from it.

There are other reasons as well, such as advanced Meson features we are
looking to make extensive use of in the future. More on that later.

[1] http://blog.nirbheek.in/2016/05/gstreamer-and-meson-new-hope.html?s
howComment=1464011145671#c8053239224284015105

== But why change at all? Doesn't the current autotools-based build
work just fine?

Technically it works fine, yes, and between all of us we have close to
a century of experience of bending autotools to our will. We're no
haters [2], but there's not a great deal of enthusiasm lost for
autotools either.

[2] although we might make a special exception for libtool

The desire for change comes from many directions.

One is the desire to provide a better development setup for developers
on Windows and Mac. On Windows the situation is particularly dire. The
truth is that Windows developers want something that integrates with
their development environment and workflow, which usually means Visual
Studio. When they debug or benchmark their GStreamer applications they
want debugging symbols to show up. They want to be able to compile
GStreamer with MSVC. They most certainly don't want to faff about with
MingW and gdb, nor do they want to spend days compiling everything.
Watching configure run natively on Windows is like watching paint dry.

With the Meson-based build, we can now build GStreamer natively on MSVC
and in finite time, either via the ninja backend using cl.exe or by
creating Visual Studio project files (experimental, see Nirbhek's blog
post mentioned further below).

The cerbero bits needed for that have not been merged yet, but for
those who do not use cerbero, it's already possible to do so using the
merged build descriptions. For those who wonder, maintaining VS
projects manually was not considered a viable alternative in the long
run, and also won't deliver the kind of development experience we're
aiming for long-term.

The other factor that prompted a desire for change is more profound:
once you've seen a better future, you can't just unsee it again.

For those old enough to remember: after cvs, svn seemed like a
revelation, but once you've mastered git you usually didn't want to go
back. This is how some of us feel now.

Meson is not perfect yet. There are some usability issues and missing
features that could be, should be and will be improved. There are most
certainly bugs (there are always bugs). The vs and xcode backends need
work. But all in all it works rather nicely already for the feature set
that we need on the platforms where we've tested the builds on. Having
been written in python is is also extremely hackable (unlike, say,
libtool) with a supportive and superb maintainer.

== Where can I learn more about Meson?

A good place to get started is the Meson website at

  http://mesonbuild.com

For those who prefer a video, check out Jussi Pakkanen's talk "Making
Build Systems Not Suck" at LinuxConf Australia 2015:

  https://www.youtube.com/watch?v=KPi0AuVpxLI&hd=1

You may also want to peruse Nirbheek's blog post "GStreamer and Meson:
A New Hope"

  http://blog.nirbheek.in/2016/05/gstreamer-and-meson-new-hope.html

and his GUADEC 2016 talk (incl. video)

  http://blog.nirbheek.in/2016_08_01_archive.html

== What's the status of the GStreamer Meson build then?

It should mostly work just fine already, at least with the Ninja
backend.

It is expected to work out of the box on Linux and Windows, with gcc,
clang or MSVC. It should work in principle elsewhere as well, but
chances are that the build files will need some fixing up for other
operating systems (e.g. macOS).

There will still be some omissions and warts in the build description
files, such as hardcoded values, missing checks or options which
haven't been ported over yet, but most of these things are easy to fix
or add.

Cerbero will also need some changes in order to support Meson,
especially with an MSVC toolchain. These have not been merged yet, but
that's something that's going to happen over the next few weeks.

There are some plugins in ext/ and sys/ which haven't been mesonified
yet. In some modules unit tests and examples still need to be hooked
up. In -bad the OpenGL detection bits have to be ported over, so
currently libgstgl and any plugins depending on it can't be built yet.

Android builds definitely won't work yet, because cerbero's ndk-build
mechanism currently depends on libtool .la files to be present to
discover all the dependencies needed for the plugins and libraries.

It will currently only work with an installed setup, but you can
install into any prefix you like of course. Work on an uninstalled
setup is in progress, but it's not ready yet. The reason it doesn't
just work with the current uninstalled setup is that Meson always uses
a separate build directory, so all the paths gst-uninstalled sets up
don't work as they used to. On the upside, the upcoming uninstalled
build will be able to build all modules in one go, which should
hopefully make for a nicer workflow for developers tracking git master.

It was merged now because it's useful and usable already in its current
state, and having it in master will make it easier for everyone to
test, contribute and beat it into shape, a process which no doubt will
take a while.

== Where are you going with all this?

For one, we're keen to add MSVC toolchain support and improve Visual
Studio support for GStreamer.

Naturally it is also our intent to gradually work towards replacing
autotools as our build system. This is not something that can be
achieved over night, not least because it affects a large number of
people in the wider community who will need to update build scripts and
setups when it happens.

Then in the longer run we'd also like to make use of some advanced
Meson features such as subprojects for external dependencies, which
will hopefully allow us to drastically simplify cerbero and provide a
much better GStreamer development experience on non-Linux platforms and
also simplify the deployment of GStreamer-based applications for third
parties.

== So when are you going to drop autotools?

One day, hopefully. But that day is not today.

== I am a Linux developer - what's in it for me?

There's always the general enjoyment of using nicer tools [3]; but
Meson and Ninja are also *fast*. In some cases much faster, e.g. on my
quad-core i7 I can configure, build and install gst-rtsp-server with
Meson in just under 4 seconds, while with autotools it takes about 17
seconds for autogen.sh + configure alone to run, and then another 10
seconds to build things (incl. tests, excluding docs).

There are ways to speed up the build with make as well of course, such
as moving away from a recursive make setup, but this doesn't help
improve the autogen/configure part and in general doesn't seem worth
the effort at this point.

The speed-ups were never the primary goal, but they are nice and they
are noticable. I also find ninja's build output (progress report) much
nicer, this way there's no noise in the output and compiler warnings
are easy to spot.

With the Meson-based build you can also just run tools such as gdb,
valgrind or perf directly on the binaries in the build directory (no
libtool!).

[3] 'The enjoyment of one's tools is an essential ingredient of
successful work.' -- Donald E. Knuth

== I am a Windows developer - what's in it for me?

Did I mention MSVC support? This is something that's still worked on
and not ready for consumption out of the box, e.g. there are no ready-
made binaries built with MSVC yet, nor Visual Studio project files. Nor
is the Meson+MSVC toolchain support available in upstream cerbero yet.
However, if anyone is feeling adventurous and wants to give this a
spin, they can follow Nirbheek's blog post "Building and Developing
GStreamer using Visual Studio":

  http://blog.nirbheek.in/2016/07/building-and-developing-gstreamer-usi
ng.html

Alternatively, it is possible to build GStreamer with MSVC without
Cerbero, but that requires all dependencies to be available already. We
know it works, but most people will not find this the easiest way.

== I am an OS X / macOS developer - what's in it for me?

You could go down in GStreamer history as the first person to report
all the things that probably don't work quite right yet on OS/X with
the Meson build!

== How can I get started with Meson?

To try out a Meson build, the new gst-examples repository might be a
good starting point:

  $ git clone https://cgit.freedesktop.org/gstreamer/gst-examples
  $ cd gst-examples
  $ mkdir build
  $ cd build
  $ meson && ninja

This should build some local test apps which you should be able to use
from the build directory without installing them first.

Requires GStreamer, Gtk+ and libsoup development headers/files to be
installed. And of course recent versions of meson and ninja.

Building GStreamer itself and the plugin modules works just the same
way, but you should only attempt this if you are familiar with building
GStreamer from source and how to work with a from-source GStreamer,
especially if installed into a custom prefix.

In any case, this just as a quick heads-up on what's happening with
this and what it's all about.

As always, any testing and contributions are much appreciated. Pop into
in #gstreamer or #mesonbuild on freenode IRC if you have any questions
or want to help out.

Happy hacking!

Cheers
 -Tim

-- 
Tim Müller, Centricular Ltd - http://www.centricular.com

Join us at the GStreamer Conference: 10-11 October 2016 in Berlin,
Germany