[Mesa-dev] [RFC] Convert mesa to automake/libtool

[Ben Gamari]

On Sun, May 2, 2010 at 12:46 PM, Dan Nicholson <dbn.lists at gmail.com> wrote:
> Brian,
>
> I'm putting forward this request completely understanding your
> position why you don't want automake and libtool in your project.
> However, I think that mesa has outgrown the static Makefiles approach
> for a number of reasons. For a project that's grown to the complexity
> of mesa, I believe you need something that is more flexible and robust
> than the current system can provide. Eric (and I think Corbin, too)
> has a branch adding automake and libtool to the mesa repo.
>
>  http://cgit.freedesktop.org/~anholt/mesa/log/?h=automake
>
> I haven't looked at it in detail, but I know Eric knows what he's
> doing as he's maintained many of the autotooled xorg modules. Here are
> some of the pros and cons I see to making this change.
>

<Uninformed rambling>

Hey all,

As a casual follower of Mesa, I too have been bitten bad builds on more than a
few occassions. That being said, Autotools makes me cringe and certainly makes
the build system less transparent to the end user. It seems that there is an
assumption that there exists only two options underlying this discussion:
Automakes, or what we have. I'm quite surprised that nobody has considered the
possibility of adopting a standard non-recursive makefile structure, as is used
in, say, Keith Packard's notmuch[1] project.

I recently reworked the build system of an internal piece of scientific
computing software with a notmuch-inspired makefile structure and the results
have been fantastic. The build is much faster, the makefiles are clearer (in
most cases nothing more than lists of rules), and the build integrity is
guaranteed (up to the quality of the prerequisite lists). Moreover, I was quite
surprised by how well suited Make was to this style of build system. When one
adds a few features of GNU Make (per-target variables), the result is quite
impressive.

> Pros:
> * AM_CONDITIONAL provides a clean way to separate optional parts of
> the build. The way that optional components are handled right now is
> pretty fragile and basically amounts to having lists of subdirectories
> being correct in the config file.

I don't see what's wrong with the ifdef directive. This seems like the right
way to deal with this in fact.

> * For all mklib's simplicity, it is inconsistent between platforms,
> doesn't handle errors and provides only a scant amount of the features
> that libtool does. Libtool provides a robust and well tested means to
> generate libraries that handles nearly all the gory details about
> generating working binaries on many platforms. I don't think anyone
> working on mklib can claim to have the same type of knowledge about
> platforms and toolchains as the libtool developers.

There's no reason why libtool requires autotools. This could be easily
integrated into a custom build system if need be.

> * Consistency in build commands for free from automake. Right now we
> have the compiling and linking commands repeated throughout the tree
> and they're typically out of sync. I've tried to keep them in sync
> before and it was a lot of effort. With automake all you really need
> to do is tell it the CFLAGS and LIBS to use and it'll take care of the
> rest.

This should be completely solved using a non-recursive build system, where make
knows all build dependencies.

> * Parallel make jobs just work. I've fixed so many of these race
> condition bugs, but they'd all just go away using automake. It has all
> the goop built in that people usually never think about.

Again, this comes for free with make having the entire dependency tree.

> * Well defined distribution for tarballs. The top-level Makefile does
> the job, but automake can make this a lot easier and more robust. It
> would also be simple to handle the generated files while also
> requiring they be included in the tarball.

Not sure what to say about this. It seems like it wouldn't be too difficult to
implement a robust tarball rule, but maybe I'm missing something.

>
> * Fast source dependencies without external tools. The makedepend
> route works, but automake handles this in a faster, more robust and
> safer manner. We get a lot of people posting to the mailing list about
> build errors were the solution is "make realclean". This would solve a
> lot of those issues.

Yeah, you still need makedepend, but I don't really view this as a problem.

>
> * Mindshare from xorg autotools. Many of the people here have and do
> work with the autotools via xorg, so it's not like this is a
> completely foreign beast.

This is true. For the rest of us who run like children whenever autotools is in
the room, it would be unfortunate.

> * Moves the burden of build tool knowledge onto someone else. I don't
> think anyone here wants to become an expert on compilers, linkers and
> ABI for multiple platforms. It become a lot easier when you toss
> things off to libtool where people are actually spending their time
> caring about them.

Again, libtool doesn't require autotools.

> * Extensive documentation available, unlike the current system which
> is pretty much ad hoc.

Documentation can be written for any build system, although in my experience, a
non-recursive build system is quite straightforward and the fact that it's pure
make means that pretty much anyone can grok it.

> Cons:
> * The abstracted nature of automake causes build debugging to be
> difficult. This requires you to train your brain not to look at the
> generated Makefile, but still it can be difficult. Fortunately, many
> of the build bugs we see today in Mesa would go away with automake.

Yep, definitely a big con for many. The source generated by autotools is
completely uncomprehensible to any mere mortal. A non-recursive build system
would get rid of the build bugs while keeping things understandable with
minimal background.

> * Using libtool means that you can't quickly hack around a fix for
> some platform. Fortunately, libtool is a lot more stable these days on
> common platforms.

Quite true.

> * The maintainer (you) doesn't like it. Not much I can suggest here
> besides that it gets a lot easier the more you deal with it. And I'd
> be happy to help you out of any jams. For xorg, Peter Hutterer has
> asked me to solve a bunch of these problems, and I can't remember the
> last time we couldn't get something fixed.

I'm not a huge fan either, but my opinion really doesn't matter here.

> * Loss of the simple "make $target" usage. I understand you guys use
> these targets to quickly pop out a build. As a compromise, we could
> turn the configs into wrapper scripts that generated the autotools
> files and ran configure with appropriate arguments. For example,
> "./configs/linux-debug && make". Or, since configure handles the
> platform parts, ./configs/debug or ./configs/osmesa.

My non-recursive makefile structure keeps the ability to make $target. It's
quite trivial to implement in fact.

> That's all I can think of. I'm sure we can continue to make the
> current system work, but I think integrating these tools would be a
> big improvement. Thanks for considering it.

Anyways, those are my two-cents. Anyone who's curious really should have a look
at the notmuch build system. I would also love to share my own implementation
(which is probably more suitable for a larger project like mesa). I think you
would be quite surprised to see just how naturally the non-recursive paradigm
fits within make. Thanks for listening.

Cheers,

- Ben


[1] http://www.notmuchmail.org/