[Mesa-dev] [RFC] Convert mesa to automake/libtool

[Dan Nicholson]

On Mon, May 3, 2010 at 2:04 PM, Brian Paul <brianp at vmware.com> wrote:
> Dan Nicholson wrote:
>>
>> Brian,
>>
>> I'm putting forward this request completely understanding your
>> position why you don't want automake and libtool in your project.
>> However, I think that mesa has outgrown the static Makefiles approach
>> for a number of reasons. For a project that's grown to the complexity
>> of mesa, I believe you need something that is more flexible and robust
>> than the current system can provide. Eric (and I think Corbin, too)
>> has a branch adding automake and libtool to the mesa repo.
>>
>>  http://cgit.freedesktop.org/~anholt/mesa/log/?h=automake
>>
>> I haven't looked at it in detail, but I know Eric knows what he's
>> doing as he's maintained many of the autotooled xorg modules. Here are
>> some of the pros and cons I see to making this change.
>>
>> Pros:
>> * AM_CONDITIONAL provides a clean way to separate optional parts of
>> the build. The way that optional components are handled right now is
>> pretty fragile and basically amounts to having lists of subdirectories
>> being correct in the config file.
>>
>> * For all mklib's simplicity, it is inconsistent between platforms,
>> doesn't handle errors and provides only a scant amount of the features
>> that libtool does. Libtool provides a robust and well tested means to
>> generate libraries that handles nearly all the gory details about
>> generating working binaries on many platforms. I don't think anyone
>> working on mklib can claim to have the same type of knowledge about
>> platforms and toolchains as the libtool developers.
>>
>> * Consistency in build commands for free from automake. Right now we
>> have the compiling and linking commands repeated throughout the tree
>> and they're typically out of sync. I've tried to keep them in sync
>> before and it was a lot of effort. With automake all you really need
>> to do is tell it the CFLAGS and LIBS to use and it'll take care of the
>> rest.
>>
>> * Parallel make jobs just work. I've fixed so many of these race
>> condition bugs, but they'd all just go away using automake. It has all
>> the goop built in that people usually never think about.
>>
>> * Well defined distribution for tarballs. The top-level Makefile does
>> the job, but automake can make this a lot easier and more robust. It
>> would also be simple to handle the generated files while also
>> requiring they be included in the tarball.
>>
>> * Fast source dependencies without external tools. The makedepend
>> route works, but automake handles this in a faster, more robust and
>> safer manner. We get a lot of people posting to the mailing list about
>> build errors were the solution is "make realclean". This would solve a
>> lot of those issues.
>>
>> * Mindshare from xorg autotools. Many of the people here have and do
>> work with the autotools via xorg, so it's not like this is a
>> completely foreign beast.
>>
>> * Moves the burden of build tool knowledge onto someone else. I don't
>> think anyone here wants to become an expert on compilers, linkers and
>> ABI for multiple platforms. It become a lot easier when you toss
>> things off to libtool where people are actually spending their time
>> caring about them.
>>
>> * Extensive documentation available, unlike the current system which
>> is pretty much ad hoc.
>>
>> http://www.gnu.org/software/autoconf/manual/autoconf.html
>> http://www.gnu.org/software/automake/manual/automake.html
>> http://www.gnu.org/software/libtool/manual/libtool.html
>> http://sourceware.org/autobook/autobook/autobook.html
>>
>> Cons:
>> * The abstracted nature of automake causes build debugging to be
>> difficult. This requires you to train your brain not to look at the
>> generated Makefile, but still it can be difficult. Fortunately, many
>> of the build bugs we see today in Mesa would go away with automake.
>>
>> * Using libtool means that you can't quickly hack around a fix for
>> some platform. Fortunately, libtool is a lot more stable these days on
>> common platforms.
>>
>> * The maintainer (you) doesn't like it. Not much I can suggest here
>> besides that it gets a lot easier the more you deal with it. And I'd
>> be happy to help you out of any jams. For xorg, Peter Hutterer has
>> asked me to solve a bunch of these problems, and I can't remember the
>> last time we couldn't get something fixed.
>>
>> * Loss of the simple "make $target" usage. I understand you guys use
>> these targets to quickly pop out a build. As a compromise, we could
>> turn the configs into wrapper scripts that generated the autotools
>> files and ran configure with appropriate arguments. For example,
>> "./configs/linux-debug && make". Or, since configure handles the
>> platform parts, ./configs/debug or ./configs/osmesa.
>>
>> That's all I can think of. I'm sure we can continue to make the
>> current system work, but I think integrating these tools would be a
>> big improvement. Thanks for considering it.
>
>
> Hi Dan,
>
> Thanks for the thoughtfully worded pros & cons.  Your arguments are
> compelling and I can see value in a switch to automake.  Just let me list my
> concerns.
>
> Some years ago someone converted Mesa to use autoconf/libtool and it was
> pretty awful.
>
> The libtool script wrapped every cc command and it slowed the build process
> considerably.  Have you done any "before/after" build time comparisons?  I
> build a *lot*.

This is a lot better nowadays. Obviously, mklib has very little
overhead. The xserver has dolt, which basically overrides libtool and
turns it into a simple wrapper script sort of like mklib on most of
the platforms you'd care about. This was a big improvement over
libtool-1.5, which many hosts have. However, a while back I timed the
more recent libtool-2.2 series against dolt compiling the xserver and
there was very little overhead from dolt.

http://lists.freedesktop.org/archives/xorg/2008-December/041271.html

All recent distros include libtool-2.2 (out since 3/2008), so it's a
non-issue for most people.

> Despite the goal of cross-platform support for autoconf/libtool, it actually
> seldom worked properly on anything but GNU/Linux.  People who tried to build
> Mesa on AIX, Ultrix, SunOS, HP-UX (including myself) were screwed and I
> wasted lots of time trying to figure out the problems.  In the end I usually
> threw my hands up in frustration and gave up. Meanwhile, the static configs
> were easy to fix and worked great for many years.

I think a lot of these issues have been worked out in recent years. I
don't know how many people try to build for those platforms on xorg,
but I can't recall the last time there was an issue caused by the
autotools for a unix-y platform. Trying to build on windows
(non-cygwin) or osx (with the "fat" binaries) seems to go wrong
sometimes, but people still do it on xorg, and I don't see hacks to
get it done.

Alan Coopersmith builds xorg for solaris consistently without problems
caused by the autotools (to my knowledge). The cross platform issues
that arise are usually code level, which isn't affected here.

I can't say this is a dead issue, but I rarely see it anymore.

> Probably 99% of Mesa users on Unix are using Linux or *BSD but I still feel
> some loyalty to that other 1% and would feel bad if Mesa no longer worked
> for them.

Certainly, but I actually would expect this to work in the 1% favor.
The autotools are used in so many packages that many people know how
to manage them and what they need to do to get a build done.

For the person not familiar with autotools and trying to build on an
obscure platform, this would obviously be challenging. But mesa
wouldn't be the only one in that boat. Take bluegene. That's pretty
obscure for most people, but it looks like all the needs to happen is
that the compiler path is set and the correct options are passed to
configure. That's just like the multitude of other autotooled packages
they probably encounter.

> Nearly every day I get email from people with questions about Mesa (how to
> build it, use it, etc.) - stuff you never see on the mailing lists.  If we
> go the autoconf/libtool route I'd have to forward all such questions about
> building Mesa to you or someone else.  I'd be counting on you/someone to
> address issues in a timely manner.  I like to try to help people with their
> problems and I kind of hate to pass the buck but I wouldn't have any choice.

Yeah, that's understandable and obviously the biggest drawback. I
would say that a bright guy like you would pick up the gist of
automake pretty easy. One of the tough issues is invoking the
"regenerate autotools" step when necessary. But stuff like automake's
--enable-maintainer-mode can make this easier.

On the other hand, I'd be happy to help out with build issues. I
usually stick my nose in on mesa-dev, but don't follow mesa-user.
Fortunately, there are a lot of people here who have working knowledge
and would be able to help out. Probably more than now, too, since
there are only a couple people who really understand how the mesa
build system works.

> There are some lesser-used drivers like OSMesa and the glslcompiler binary
> that I would want supported.  Can you take care of that too? I'm afraid that
> the nooks and corners of Mesa (like those two) which aren't of interest to
> you or Eric will get ignored.

In the past I have tried to keep as many of the configurations
(including OSMesa) building as possible. This is where having a
standard system really shines, though. Automake will take care of the
building no problem. It's just another library or another program. The
challenge really is figuring out the autoconf goop (and automake
conditionals) so that it gets turned on at the right time with the
right options. We've already been handling that stuff, though.

> When the person who contributed autoconf support years ago disappeared there
> wasn't anyone left to take his place so I quickly and happily ripped out all
> that stuff and set up the static build system.  I doubt that'd happen again,
> but that's what happend last time.
>
> I dread being in situations where something's broken and I can't fix it
> myself.  That's one of my main issues with the proposed change.

Understandable. As long as xorg is using the autotools, I expect it
wouldn't be a problem finding people to fix issues. There are a couple
people I can think of who would happily run through it and make
everything clean and consistent, too. On the other hand, you'd
probably just need to learn it if you wanted to be self sufficient.

Obviously, this is your project, and you're the maintainer. I wouldn't
want to force something on you that would become unmaintainable. So, I
would certainly help and try to make an effort to find someone to help
out or suggest an alternative if the future changed.

> So anyway, I just checked out Eric's git tree and the automake branch.
> I ran "./autogen.sh":
>
> $ ./autogen.sh
> autoreconf: Entering directory `.'
> autoreconf: configure.ac: not using Gettext
> autoreconf: running: aclocal -I m4
> aclocal: couldn't open directory `m4': No such file or directory
> autoreconf: aclocal failed with exit status: 1
>
> Off to a flying start, huh?  At this point I have no clue what's wrong.  If
> this were Mesa/master I'd be totally stuck and feeling helpless (as I
> usually do when I try to build X.org).  Maybe Eric's branch isn't in a
> buildable state.  Is there another tree/branch that I should try out?

:) I haven't actually tried Eric's branch. I just wanted to start the
conversation before spending a bunch of time working on the idea. I'll
try to check out Eric's branch in the next couple days and get it
working (at least for the default build).

--
Dan

P.S. I didn't mean to pile this on at the same time as Dave's
development model thread. I had this drafted a while back and then saw
an email about the generated sources and decided the conversation
needed to happen. Keep up the good work. :)