Depending on external libraries

[Egbert Eich]

Daniel Stone writes:
 > 
 > I do not believe this to be the case. It just causes heaps of heartache
 > for completely hypothetical situations when, say, there's a security bug
 > in zlib or something equally improbable. If you depend on a specific
 > version because of API reasons, talk to upstream! If one part of the API
 > must be absolutely rock-solid, ask them to keep compatibility wrappers
 > around.

This depends on upstream being open to your arguments.
This may not be the case all the time with all projects.

 > 
 > It just causes confusion, and it's a PITA from a distributor point of
 > view (bugs in Xrender that were fixed 4 releases ago, but still aren't
 > in XFree86, for instance).

Right. However people here are mentioning the 'distributor's point of
view' a little bit too often and neglect the 'developer's point of
view'. I as a developer like a stable environment around me. One where
I don't have to worry about all kinds of dependecies when I update
the tree I'm working in to a later version.
I'm not saying that these problems cannot be solved. However an argument
that was often used in favor of the modular system was:
     'We can much better detect if changes in the API break things'
We discussed at length what should be done to keep Xserver internal
APIs stable. 
Now I have discovered that the API stability has not been an issue
for some of the modular lib packages:
When upgrading Xft a newer version of fontconfig was required.

The only 'external' dependency we had (a dependency to SW not hosted
on fd.o) was freetype2.
Both Xft and fontconfig didn't build with the latest version of
freetype2 (although earlier versions - ie. the ones in the XFree86 
tree - did). The changed include file scenario of freetype2 has
been known for a long time, therefore we cannot claim the freetype2
team has been unreasonable and changed things behind our back.

Until yesterday the latest released version of libXft depended on a
version of fontconfig that was still in a prereleased state.

Therefore, if we still need to grab the latest pices of everything
when we want to update one single piece we are better of shipping
a giant tarball. Then the user deosn't have to worry which pieces
belong together.

 > 
 > > > Asking people to pull such things from an external source may not
 > > > be too much to ask if we detect the missing pieces reliably and
 > > > early in the build and instruct the user where to get them. 
 > > 
 > > IMHO asking people to pull these pieces from an external source is just 
 > > asking for trouble down the track. There is no way to properly certify 
 > > that the 'system' or 'component' you are interested in is really going to 
 > > work unless the external version works completely *unchanged* for your 
 > > component (ie: no patches required from an official release you can 
 > > download from an external server).
 > 
 > Their distributor should also provide them: we should be doing this if
 > there is a clear expectation that your distributor will provide a
 > certain package (e.g. zlib). Beyond here we're possibly talking about a
 > monolithic/modular architecture, which seems to be too much of an
 > emotionally-charged topic to be objective.

Not really. I'm comparing the arguments brought up in favor of the
modular build with the reality I'm seeing.
Other than that I don't understand what you are trying to say in the
previous paragraph.

 > 
 > > Note also that the policy of including all the source to build the 
 > > necessary pieces does not necessarily infringe on the idea of a modular 
 > > build system either. You only need to keep around the pieces required by 
 > > each modular part of the system as necessary. The key thing is to make 
 > > sure stuff is broken up properly and that stuff is properly integration 
 > > tested together as a unit.
 > 
 > Right, but it defeats half the point, IMO. I will personally not be at
 > all happy with any non-monolithic tree that includes external libraries.
 > From a distributor point of view, it's caused me a hell of a lot of
 > pain. If you depend on external libs, why not just tell people where to
 > go get them?

This is again the 'distributor's point of view'.
Did you ever get a system for testing with a OS preinstalled
of which you didn't know much more that it contains some basic
build tools? And did you ever have to build and test an X environment
on such a system? With the monolithic tree this is easy!
With the autotooled modular tree this can be a PITA!

If you are lucy configure will tell you which packages are missing
or need a later version. This happnes one package at a time, therefore
you will need to run configure again for each package you put onto
the machine.
If you are unlucky configure doesn't complain but your build brakes
with some 'undefined  symbol' leaving you wondering which of the
external dependicies (which you may not even understand completely)
you have to update to which version to fix the build.

In any case if you are not allowed to install software on this
test system beyond your working directory you are left wondering 
where you should put those pieces so that they are found instead
of the installed ones.

I do realize that 'package maintainers' don't need to worry about
all this. However they are not the only ones who are affected.

 > 
 > > IMHO it would be more prudent for a vendor wishing to see new 
 > > functionality in say a later release of FreeType deployed in their 
 > > distro, spend the time to merge in the latest version of FreeType and 
 > > iron out the issues in the *main tree* rather than just for their distro. 
 > > That way everyone benefits from this work, and there will be a lot less 
 > > duplication off effort.
 > 
 > We're talking about forking every single library for no reason.

Freetype is a bad example. I'm not worried about a few well defined
external dependencies on packages that are maintained elsewhere
especially if the API is kept relatively stable and we take some care
that we can build against earlier version of this API.

However the tight (version) dependencies among the libraries that
are not maintained elsewere invalidate the argument for modularization.

 
 > 
 > > After all isn't one of the reasons this X.org version of the X server got 
 > > started was to avoid all the duplication of effort among the distro 
 > > vendors because they couldn't easily get their patches and changes back 
 > > into the core system? If that barrier is significantly lower, there is no 

Generally to give a larger group an opportunity for participation so
that work can be distributed on more sholders.
This of course also applies to people working for distros.
It doesn't mean that we support flooding the code with distro specific
code.

 > > reason why the copies of the library code inside the CVS tree cannot be 
 > > easily updated by those interested in new features and functionality 
 > > provided by updated versions of external products. Upgrading for the sake 
 > > of upgrading is a pointless exercise - unless there is some tangible 
 > > benefit to using a later version of FreeType or some other library (bug 
 > > fixes, performance enhancements or new features), using it when it may 
 > > cause problems seems a little silly.
 > 
 > Duplication of libraries seems a lot like duplication of effort to me.
 > ;)
 > 

This is correct. The theory behind the xc/extra directory is that
new versions of external libraries could be dropped in 'as is'
an Imakefile existed (not in xc/extras) to embed the build into
the context of an X-style build.
In principle these libs are provided for convenience only - nobody
seems to realize this.

Egbert.