[Xcb] [ANNOUNCE] xcb-util 0.3.9

[Josh Triplett]

On Tue, Jun 05, 2012 at 10:03:44PM -0700, Jeremy Huddleston wrote:
> On Jun 5, 2012, at 6:35 PM, Josh Triplett <josh at joshtriplett.org> wrote:
> >> Take libXi for example, since it's probably fresh in most of our minds.  With Xi2, libXi added a ton of new functionality and remained backwards compatible.  Thus, the minor version of the library was bumped (http://cgit.freedesktop.org/xorg/lib/libXi/commit/?id=0d19a3ec942aedf5432a9bda1e80f29f7186ce5b) from 6.0 to 6.1.
> >> 
> >> As far as I can tell, though, symbols are not annotated with "available in 6.1 and later" so the only difference is the minor version bump, and as you said, Linux/ELF essentially ignores this ... so in this case, there is no *practical* difference between going forward with an added symbol and going backwards with a removed symbol.  To make this a bit more clear (words fail me, sorry), consider:
> >> 
> >> case 1:
> >> libABC.0.0 contains symbols a, b, c
> >> libABC.1.0 contains symbols a, b
> >> libABC.1.1 contains symbols a, b, c (c was added back in but not annotated as new in 1.1)
> >> 
> >> case 2:
> >> libABC.0.0 contains symbols a, b, c
> >> libABC.0.1 contains symbols a, b
> >> 
> >> In case 1, libABC.1.1 and libABC.0.0 are identical.  Since Linux/ELF
> >> ignores minor version, there is no effective difference between
> >> linking against libABC.1.1 and running with libABC.1.0 in case 1
> >> compared to linking against libABC.0.0 and running with libABC.0.1 in
> >> case 2.
> > 
> > I agree with your statement that from a functional standpoint this holds
> > true: the linker doesn't seem to enforce the minor version rule, so you
> > can build against a newer library and run with an older one, or vice
> > versa, as long as the major version matches.  The linker will complain
> > if you use a symbol that it can't resolve, though.
> 
> As it should (well unless the symbol is weak and can be checked for at
> runtime).

True, though for most symbols that doesn't make sense, since you'd have
to write things like "if (!xcb_useful_function) aieee();". :)

Better to use symbol versioning or similar to let the dynamic linker
tell you at the start of your program that a symbol doesn't exist.  Or,
if you really have written your program so that you can cope with the
absence of some functionality, consider either using dlopen/dlsym to
make that explicit or otherwise having a way to easily verify the
functionality you need without having to test every symbol for NULLness.

> > Linux distributions primarily want to avoid the scenario in which the
> > dynamic linker considers a library version acceptable but the program
> > requiring that library crashes or otherwise breaks at runtime.  
> 
> I think that's a fairly universal concern beyond just Linux distros.

Agreed; I primarily mentioned this scenario to distinguish from the case
of "the dynamic linker complained and refused to run the program due to
a missing symbol", which still amounts to failure but in a controlled
way.  I didn't intend this as "non-Linux systems don't care about the
runtime-crash scenario", but rather as "Linux systems don't enforce the
minor version requirement because enforcing the presence of symbols
avoids the runtime crash scenario for all cases except changing the
interface of an existing symbol".

> > That
> > doesn't happen with the addition or removal of symbols, because the
> > dynamic linker will attempt to resolve those symbols and complain if it
> > can't; in that scenario, the rule of bumping the major version when
> > removing functions represents a "don't do that" convention to avoid
> > having users surprised by programs that suddenly refuse to run with a
> > library that claims compatibility; upgrading to a newer version of a
> > library with the same major number should never cause existing programs
> > to stop working.
> 
> Yes, I thoroughly agree with that, but I also believe that applications
> built against a newer version of a library should be *capable* of running
> on older versions of that library (with the same major version) and that
> API should *almost never* be removed (deprecated yes, removed no) to allow
> the best portability possible.

I can see the argument for the former, in that it potentially makes
sense to allow a program built against a new library to run with the old
if it uses none of the new functionality.  I agree with the latter, but
I'd also say that when removing API, the library major version should go
up to express the incompatibility of old programs with the new library.

> > In particular, the minor version serves as a hint to the programmer that
> > if they link against libABC.so.1.1, they might or might not successfully
> > run against libABC.so.1.0, depending on what symbols they used.
> 
> IMO, that should be annotated in header files in a way that allows those
> symbols to be weak linked and checked for at runtime (and thus go down an
> alternative codepath if unavailable).

Not unless that gets wrapped in some kind of interface that avoids the
need to check all used symbols against NULL before using them; I'd
prefer to make that the dynamic linker's job.

> >  On the
> > other hand, a programmer expects that a program linked against
> > libABC.so.1.0 should *always* work with libABC.so.1.1, but won't
> > necessarily work with libABC.so.2.0.  
> 
> Well, it won't *EVER* work with libABC.so.2.0 because the dynamic linker
> sees it as a completely different library.

True.

> > Removing or changing symbols
> > breaks that assumption; adding symbols doesn't.
> > 
> > Libraries typically introduce symbol versioning to handle more complex
> > cases that the linker can't catch on its own, namely a *change* to an
> > existing symbol, such as by adding a new parameter to a function, or
> > more subtly by changing the layout of a data structure used by a
> > function.  In that scenario, the linker will see no problem, but the
> > program will break at runtime, precisely as if you cast a function
> > pointer to a different type with different parameters and tried to call
> > it (because, effectively, you did).  Symbol versioning lets you maintain
> > ABI (though often not API) compatibility in this case, by having old
> > programs use a compatibility symbol that knows how to handle the old
> > calling signature.
> 
> Yeah, we essentially have that same mechanism in place, usually for
> cancelable/non-cancelable variants, legacy vs UNIX-conforming variants,
> changes to 'struct stat', ...

Any notable differences between ELF symbol versioning and OS X dylib
symbol versioning?

> > However, the details of how much the dynamic linker enforces and how
> > much just occurs by convention don't change the general rules for how to
> > set library major and minor versions: always bump the major version if
> > you remove symbols or change the meaning of existing symbols.
> > Exceptions to that rule lead to unexpected breakage, even when
> > attempting to claim that a symbol doesn't represent public API, and such
> > exceptions should only occur with careful consideration of the breakage
> > they'll introduce.
> 
> I just disagree with this point.  If a symbol exists just for internal
> consumption but is exported (yes, the developer should've used visibility),
> then I don't believe removing it warrants a major version bump.

I didn't say to *never* remove functions even when you consider them
entirely internal; I just suggested that functions considered "internal"
but exposed for public use have a nasty habit of becoming publically
used, in ways that people don't complain about until they break. :)

- Josh Triplett