[systemd-devel] bus driver on kdbus

[Lennart Poettering]

On Mon, 14.04.14 10:48, Ryan Lortie (desrt at desrt.ca) wrote:

> > I will now remove the entirety of the driver daemon, not just
> > AddMatch/RemoveMatch as all but two of those calls are subject to
> > similar (theoretical...) races.
> 
> It's always been somewhat clear to me that kdbus is not a
> reimplementation of dbus, but rather something completely different. 
> The extent of this difference is starting to get to the point where it's
> distracting.

Au contraire. I am pretty sure that with handling the driver client side
makes things much simpler and closer to classic dbus1. I mean,
previously we already had this weird confusion that some of the driver
apis where handled client side (where the kernel's notifications would
be converted into NameOwnerChanged), while others where handled by
driverd.

With moving processing of these messages to the client side everythwing
becomes neatly uniform. Sending a message to the bus driver is just an
alternative way for executing an ioctl. The execution order and
semantics are then *exactly* the same on dbus1. We pretty much get as
close to dbus1 semantics in this regard as possible. The only difference
between dbus1 and kdbus here is that in dbus1 you encode your requests
in a marshalled bus messagend the other time in a simple ioctl. But the
execution is scheduled exactly and precisely the same way as on dbus1,
and that is a great boon for compatibility.

> I understand that you're walking a fine line here between the existing
> large userbase of D-Bus and the requirements of simplicity of kernel
> code, but it's starting to feel like this is "D-Bus" only in name -- a
> name taken as a way of adopting the current userbase.

Well, Ryan, you asked for adoption of gvariant for everything, so you
require us to remarshal for all messages. Where's the precise difference
between remarshalling payload messages to/from gvariant, and driver
messages to/from ioctl calls? Ultimately, it's the exact same thing,
whether you marshal into gvariant and call sendmsg() or whether  you
marshal into some struct and call ioctl().

If you have an issue with different marshallings, then we can certainly
drop the gvariant thing and speak only dbus1 marshalling again, but I
wasn't aware that remarshalling was something you had an issue with? ;-)

If we can intellectually separate semantics from marshalling, then the
new scheme really boils down to that the execution scheduling is simply
turned into exactly what it was in dbus1, so we get closer to "D-Bus",
not further away.

> > driver calls on the client side, but I don't see any other way that
> > fixes the race and was somewhat clean to me.
> 
> David and I discussed some possibilities in Nuremberg, including
> treating communications with the bus driver as special -- indeed, they
> already are, due to the "on behalf of" functionality.  When a call is
> pending with the bus driver, other messages from the process that sent
> that message would be frozen.  We could even improve the "on behalf of"
> API by limiting it to target clients that are currently being served by
> the driver.  This could possibly be implemented by explicit queue
> control APIs in the driver (since we already have other APIs there that
> impact the messages that a client sees, for example).

With the new scheme we can simply drop the on-behalf-of thing,
everything becomes much simpler, we need no further synchronization, the
right synchronization happens automatically, because we schedule things
exactly as we did on dbus1.

Let's not complicate this all by inventing some weird inter-peer
synchronization mechanisms, and turning the bus driver into some magic
thing. By doing the driver handling client side, we actually make it
less magic, since talking to the bus driver is reduced to simply using a
different marshalling+syscall, and that's it.

> We could also add proper support for match rules to the kernel...

No. We are not going to marshal/demarshal things in the kernel. This is
a total no-no. 

> I haven't forgotten our discussion about the specifics of how hashing
> works for adding bits to the bloom filter and how we should design for
> change.  You were able to convince me that we could do upgrades, but
> this argument relied on my not having to implement the specifics of the
> receive-side hashing in the client library, relying instead on the bus
> driver to do it for me.  I'm no longer convinced that we can handle
> additions to the specification in a nice way (ie: without requiring
> lockstep upgrades of all of the client libraries).

Hey, we devised a scheme how we can do bloom filter upgrades, and how to
do this in a compatible fashion by uploading multiple filters per match,
and simply checking the oldest match against the message filter, that is
as new the filter is. If you follow what I mean.

We have a scheme for upgrades, and it's actually relatively easy to
implement. And it is followed by kdbus and sd-bus.

This problem is solved, I really don't see why you need to reopen this.

If you want i can put together a "reference implementation" of the
hashing logic. You'd just pass a list of tokens to add to the hash
vocabulary, and we return you the match hash. You could then drop that
into your sources, and you can update it when we do another revision of
the protocol, and all that without breaking compat.

Such a reference implementation would be < 50 lines of code, plus the
siphash sources. I really don't see why this is necessary though, the
code is close to trivial, it's certainly much simpler and easier than a
lot of other facets of dbus.

> > Note that in our new sd-bus.h API we won't do any translations for
> > this. We simply expect people to invoke our high-level C APIs rather
> > than construct bus calls for the driver interfaces. We can do this,
> > since we our C library interface is entirely new anyway. This means that
> > the bus driver interface is thus kinda bound to using the AF_UNIX
> > transport. And where the kdbus transports is used ioctls are the way to
> > go.
> 
> This really gets to the crux of my increasing annoyance.  For you, you
> have a clean slate and a fresh new API.  You don't have any promises. 
> You can tweak your API to suit the chosen implementation.

You know, I wrote the proxy for this, I know pretty much how it is to do
all these things client side, believe me.

> For the rest of us, it's not so easy.  You're imposing something on us
> that's very much *not* D-Bus and asking us to wrap it up and pretend
> that it is, and that it's just the same as it was before.  We're left
> holding the bag of the (increasingly) ugly hacks required to accomplish
> this -- and these ugly hacks have to be repeated several times, by each
> client library.  We could all do nothing and just fall back to the old
> protocol, but I'm pretty sure that would be missing the point of this
> whole exercise.

You are highly unfair. It's really now a question of marshalling to
different things conditionally:

foreach message to send:

        a) if destination == org.freedesktop.DBus → marshal to gvariant, send with sendmsg()
        b) if destination != org.freedesktop.DBus → marshal to some struct, send with ioctl()

foreach message received via recvmsg():

        a) if sender == kernel → marshal from struct
        b) if sender != kernel → marshal from gvariant

And that's really all that it is now: different encodings depending on
one special-cased destination. Beyond that we are closer to the dbus1
semantics than we *ever* were.

> I agree that the bus driver was inelegant from the standpoint of
> serialisation on startup, but this problem could have been solved
> another way: by putting this stuff in the kernel in the first place.

That's never goingto happen. We are not going to demarshall things in
the kernel. I know that Canonical wants that for its app sandboxes, but
it's just against the whole excercise. This is not going to fly. We do
direct userspace-to-userspace copies, and that allows us to do
single-copy IPC. We are not going to fuck this all up to just get back
to the old 10-copy design of dbus1.

> If you want to replace D-Bus, then you need to replace D-Bus.  What
> you're creating is looking less and less like D-Bus and more like a
> generic communication framework that one could use (given enough hacks
> and a strong stomach) to make something that's approximately compatible
> with D-Bus.

This is closer to dbus1 than it ever was. 

Let's turn this around: i can see that with your gdbus design requiring
us to remarshall everything to gvaraint makes things a lot easier for
you, but it puts the work on us to do the remarshalling. If we now
conversely ask you to smply remarshall the driver msgs differently, then
you tell us that that wouldn't be dbus anymore... Jeezus...

Lennart

-- 
Lennart Poettering, Red Hat