[systemd-devel] feature request: dlopen

[Luke Kenneth Casson Leighton]

On Wed, Feb 18, 2015 at 10:00 AM, Lennart Poettering
<lennart at poettering.net> wrote:
> On Tue, 17.02.15 20:24, Luke Kenneth Casson Leighton (lkcl at lkcl.net) wrote:
>
>>  i note that there was announcement recently that the systemd team
>> 'listens to users', so i am taking you at your word on that.
>
> Hmm, I am not aware of such an "announcement".

 somewhere there was a blog post i read of yours where (probably
garbled) i got the impression that that was the case.  a possible
explanation is that  i generally perceive "blogs" to be
"announcements".

> I generally listen though, but I don't always agree.

 that's good to hear

> I particularly don't listen to badly researched conspiracy theories.

 good!  i trust you're happy to discern the difference between such
and an attempt to understand - from a cross-project perspective - what
the heck is going on here.

i'm applying my reverse-engineering hat, here (normally reserved for
assembly code or network systems) to the situation surrounding
systemd's... rejection, shall we say.  as i mentioned to tobias's post
a few minutes ago, the situation - the bad feeling that's got
sysadmins and experienced unix users worried and has made enemies out
of decades-established acquaintances in high profile GNU/Linux distros
- is completely without precedent in the history of free software.

 i don't have all the answers, but i _can_ tell that something's
drastically wrong.


>> so i'm not going to "protest" - i'm going to try a different approach.
>> i'd like you to look at this list of debian packages that are
>> dependent on libsystemd0:
>> http://lkcl.net/reports/removing_systemd_from_debian/list_of_libsystemd0_dependent_packages.txt
>
> Well, I doubt this is correct,

 lennaaaart, smacked wrist! :)   greg's answer (asking me to provide
the information on how that file was created) is the exemplary one.
apologies that i wasn't keeping up-to-date with the incoming messages,
otherwise i would potentially have provided you with the commands to
replicate the above list, earlier, so there would be no doubt.

 ohh ok i'll work out the commands rather than do it by hand like i
did for that list last week, here it is:
lkcl at bigmac:~$ apt-rdepends -r libsystemd0 | grep "Reverse Depends" |
cut -f2 -d: | cut -f2 -d' ' | sort | uniq | wc
Reading package lists... Done
Building dependency tree
Reading state information... Done
   4583    4583   70416

the other one (which provides the list of *immediate* dependencies),
is "apt-cache rdepends libsystemd0"



> and even if it was, I am not sure what
> systemd upstream has to do with it.

 i went over this in the [now that i re-read it, very long...
sorry...] reply to tobias.

> Convince the upstream developers
> in question not to link against systemd's libraries, or convince the
> distros not to package it like that.

 well, you could provide hints in the documentation (and force them to
be read by deliberately changing the API)

 and you could provide an example by leading the conversion.  for
example, i understand that you're the developer behind portaudio?
that would be a good place to start, showing people how to begin the
process of dlopen()ing libsystemd0, documenting it well so that it was
easy to follow.


> We just offer a library, which
> apparently is interesting to people to use, but whether they use it is
> really not up to us.

 well... they've used it, alright.  so much so that there isn't a
single major GNU/Linux distro left, nor a major GNU/Linux desktop
environment, that may be used *without* systemd or a
systemd-compatible component.  xfce4, lxde, kde, gnome - they're all
now critically and exclusively dependent on systemd.  clarification,
there: i'm aware for example that KDE5 now uses the dbus logind API,
so the FreeBSD team are considering writing a replacement logind
rather than fishing through the old commits that removed the previous
code.

 and that's the problem in a nutshell - the chain if you like of
decisions that have got us to this very bad situation:

 * distros are forced to follow suit on upstream decisions, without
consulting what any other distros do
 * distros are (with the exception of gentoo, freebsd and macports as
they are all source-based) forced to hard-code the best possible
compile-time options that are *available* to them
 * the upstream decisions on a per-app basis have been made "in
isolation" without a proper across-the-board cross-project analysis.
 * the development of systemd has been done in the perfectly
reasonable way of using fedora as the proving-ground... in isolation
from other distros.

 end result: systemd is now *the* de-facto PID 1 across all major
GNU/Linux distros in the space of something like under a year, with
all other work almost completely locked out and being forced to be
"compatible" with systemd in some way (via replicating the API in some
way, by forking older versions of systemd, writing
function-for-function API-compatible equivalent d-bus services which
is *really* hard to get right, and so on).


> Also, what precisely is the problem with having a dependency on that
> library? If it's pulled in and nothing else of systemd, where
> precisely is your problem? It's just code lying around, and there is
> tons of stuff like that.

 unfortunately, on all the major GNU/Linux distros, the correlation
between "libsystemd0 being present" and "every other PID1 management
system being excluded from consideration" is near 100%.

 the detection tobias mentioned that it may have been *intended* that
libsystemd0 provide, of remaining dormant if systemd is not detected
as PID1 should signal to the upstream developers that they should
*permit* alternative PID1s  to run, but in reality the upstream
developers have simply ripped out all alternative code - not even
provided a compile-time switch to support the previous code.  the KDE5
developer recently removed the code that used console-kit (etc. etc.
whatever it was) and replaced it with a d-bus call to logind, for
example.


> If you think that code we wrote is really evil shit, that should never
> touch your system, then I figure it's too late. The folks involved in
> systemd touched a lot more of your system, outside of the systemd
> umbrella, our code is in much of the stack, in the desktop and in the
> kernel. If you want to avoid using the code we wrote you'll have a
> really hard time.

 there are people who are working actively on doing exactly that.
they're prepared to do what it takes.  personally, i'm gearing up to
become the maintainer of depinit (i just got it compiling on 64-bit,
last week).

the reasons why (their motivations for doing so) are i believe
irrelevant.  they have the right to consider doing that, and as tobias
points out, they should be encouraged for providing competition from
which the systemd team could learn.  collaboration at its best.

and *importantly*, provide a backup choice in case something goes
wrong with systemd.  hit-by-bus scenarios, monoculture development
leading to ecosystem-wide malware exploits across multiple GNU/Linux
distros, that sort of thing...

 ... except the rapid and wholesale exclusive adoption of systemd
across the horizontal (isolated) layer of upstream and the vertical
(isolated) layer of the distros *isn't allowing for any competition to
grow*.

 regarding the "evil shit" innuendo, amusing as it is, that i feel is
a red herring.  i'm aware for example of several independent technical
analysis of the code quality of systemd have encouraged others (the
author of uselessd being one example) to make it much more
cross-platform portable (he took version 209 i believe as the starting
point).  that's a good thing: the end result is that the design that
you went to a lot of trouble to put together now has a [partial]
implementation that is of very high portability quality so that it
could even conceivably be installed on FreeBSD or god help us all
Windows or ReactOS... but this is all irrelevant.

 everyone - the GNOME team, the KDE team, the cups developers - they
all work (very very hard, and very competently, it has to be
emphasised) in isolation.  the distro maintainers have been presented
with the fait-accomplit decisions of the upstreams.  _they_ work in
isolation (again, being extremely competent in their dedication to
their chosen tasks).

 and suddenly... without anyone noticing... there's no choice left but
to use systemd.

only in the far corners of the software libre community - the outlier
GNU/Linux distros and the BSDs, GNU/Hurds, Minixes and real-time OSes,
in the source-code-based OSes such as gentoo and the embedded ones
such as openembedded, openwrt and buildroot - has systemd *not* become
the exclusive de-facto PID1, where these distros are currently facing
the really hard choice of having to stop following upstream, stick
with older code temporarily whilst they scramble (e.g. with gentoo)
under enormous pressure of their users and of security fixes
potentially having to be back-ported or worse put on hold, to develop
alternatives under less-than-ideal circumstances applications such as
eudev and uselessd.

 the wholesale near-exclusive adoption of systemd _really_ has left a
lot of people with some very bad choices, and a hell of a lot of work.
as the people responsible for developing systemd, i feel that it's up
to you to do what you can to minimise the amount of effort that other
people are being put to, as well as to help reduce the enormous
tension building up right across the entire software libre world, as
best you can.


>> as i am partly writing to a public audience who may not be so
>> knowledgeable, please excuse me for describing that the advantages,
>> are, as you know, that a pre-compiled package may, at runtime, detect
>> what is available to it and use it. it may even be configured via a
>> config option to disable the use of that functionality at runtime even
>> if the dynamic library is present.  you know this.
>
> dlopen() is something the programs *using* a library must make use of,
> it's not up to us, the providers of the library.

 there are ways in which you could provide guidelines, examples, and
even deliberately change the API as an [otherwise purposeless] one-off
(the list of apt-cache rdepends immediate dependencies shows appx 100
packages, each with maintainers, so it's not like 1 person would be
required to do all 100 upstream conversions).

 i recall andrew tridgell describing to me in 2000 the best way to
design libraries, i'm sure you're aware of it.  bless him he thought i
was a such a complete novice, but i'd been reverse-engineering NT
kernels and had encountered exactly the principle he went to all the
trouble to tell me about.  i didn't speak up, because i actually
wanted to hear what he had to say, in case it contained anything i
didn't already know... ah.. i am so silly sometimes, not correcting
people's impressions... ehh never mind... where was i :)

 the method was basically to use a vector table [struct containing
higher-order-functions], publishing *only one* actual function (one
publicly exported symbol), and that function took *another* vector
table containing a list of all the functions - *all* of the functions
- that the library would ever need.  it was *not permitted* to use any
other functions.

so the [one] exported function would return a dynamically-allocated
instance of the vector table, and the contract was that it would never
call any function other than the ones it was supplied with.  the best
implemented libraries would allow you to independently call that [one]
exported function multiple times, with a different parameter
containing a *completely different* set of functions that that
instance should use, and the two "instances" should, under no
circumstances, interfere with each other.

deployment of this technique basically means that there are guarantees
about isolation between different components, and its use in the NT
kernel means that code reviews, changes to APIs and so on are all much
easier.

if a library is designed in this way, then dlopen() need only be
called once: to obtain that all-important one publicly exported
function that returns a struct that points to all available functions.

btw as you probably know (but i will outline for the benefit of people
reading this through archives in the future, hope you more experienced
people don't mind me doing this) you don't *have* to pass in a vector
table containing the functions that the library should [exclusively]
use, and in fact the majority of libraries that are out there *don't*
follow this practice... it just means that there's less guarantees,
and code analysis (unintended side-effects of modifying or improving
the lbirary) is much harder.... but that tends to be the norm.

if by some amazing chance libsystemd0 is *already* designed along
these lines, then i will be very embarrassed and will go have a good
laugh for a bit and get back to you, sharpish / sheepish or both.

if not, then you have an opportunity to make it clear to the upstream
developers the *real* intent behind why libsystemd does its PID1
detection, namely that the applications should be making *run-time*
decisions about what PID1 init system to be using, by preparing some
documentation, converting the code over thus forcing all upstream to
convert to the (recommended) method of dlopen()ing, all at once.

or not.  or something else (better than the above idea).  and you _do_
have the choice to allow this situation to perpetuate... along with
the hate-mail, the fighting, the vitriol, the burn-out and worse...

anyway, if you got this far, i must apologise for such in-depth
responses.  i do that... then i go away, never to return until another
exceptionally-complex cross-project issue arises.  once i know that
either the situation is truly hopeless and would be made worse by me
being in the story, or that i've done my bit by making the
consequences clear and you're happy but firmly made it clear that no
action is going to be taken *despite* the warnings i've given, or i
know that, as the competent team that you are, you have listened and
are planning on taking action, i'll go away happy.

i leave it up to you to decide which that should be.

l.