Starting the kdbus discussions

[Lennart Poettering]

On Fri, 03.01.14 19:55, Simon McVittie (simon.mcvittie at collabora.co.uk) wrote:

> This discusses "simple types" and "trivial types". D-Bus has no such
> things: as far as I can tell, your "trivial types" are our "fixed types"
> (all the numbers including 'y' and 'b', plus the fd type 'h'), and your
> "simple types" are our "basic types" (fixed types, plus the string-like
> types 'g', 'o' and 's').
> 
> If your use of different terminology is accidental, please rename; if
> it's deliberate, please define your terms (perhaps in terms of the ones
> in D-Bus, or by copying text from the D-Bus Specification).

It's only half accidental. I tried to stay away from the term "fixed
type", since in GVariant "fixed size" types cover more than "fixed"
types in dbus1, and I didn't want to create more confusion between the
two. (Less confusion by sloppy terminology, eh? ;-)). In gvariant
structs of of numbers are fixed size too, and so are structs of strutcs
of numbers and so on...)

> > Instead, you will only find PAYLOAD_OFF and PAYLOAD_MEMFD
> > items. The former contain an offset and size into your memory mapped
> > pool where you find the payload.
> 
> Relative to the beginning of the message, or relative to the beginning
> of the pool?

The message. (Though this actually changed just a few days ago, before
it was actually the pool. But making it relative to the message has the
benefit of making copies into multiple pools simpler since ech message
will be bitwise identical in those pools).

> > To allow clients to subscribe to specific subsets
> > of the broadcast matches we employ bloom filters.
> 
> A behavioural difference that I think it's worth mentioning: in
> dbus-daemon, the rule "broadcasts are available to anyone on the bus,
> with no security restrictions" is a convention for the configurable
> policy, but in kdbus, it's unavoidable technical fact. I don't think
> that's bad - on the contrary, if someone told me they were actually
> locking down readability of D-Bus broadcasts, my first reaction would be
> "you get to keep both pieces" - but it's a difference.

Well, the kernel-enforced minimal policy (i.e. the one that just
controls delivery from/to names and ownership of names) actually does
apply to broadcasts too. However, as the kernel will not do any further
access control (i.e. not do methods/interfaces/paths), you might leak
things if you broadcast things. Also, delivery is probabilistic,
spurious deliveries are possible. THis is certainly something to
document clearly. 

> > If the first argument of the message is a string, "arg0:" suffixed
> > with the first argument.
> 
> When you say "string" here do you mean type signature 's', or do you
> mean any string-like type? It'd be good if object paths were subject to
> arg0 processing (at least for the slash-separation case), and I don't
> see any particular reason not to extend at least arg0: to signatures.

Well, the dbus spec for arg0 matches currently explicitly supports this
only for strings. Doing this for paths and signatures as well would be
trivial though, but I didn't want to extend the spec here just by
passing by, since it's probably almost never useful...

That said, I'd be quite interested to support "as" fields for this as
well. We are planning to move udev to kdbus as transport for
notifications, as soon as kdbus is stable enough for that. libudev
allows monitor matches for "tags" which devices can have. To allow
matching against tagged devices we'd hence like to include all entries
of string arrays in the arguments in the bloom filter, so that clients
can easily test against this.

> (These instructions for how to form the bloom filter are part of the
> kernel: transport for D-Bus, not part of the underlying kernel
> functionality, right? The kernel doesn't care?)

The kernel doesn't care about the bloom filter much. The identifier is
called "bloom", but other than that the kernel doesn't know much about
the contents of the match mask or the message filters, it will just
stupidly test them against each other bitwise. In fact, userspace even
decides on the parameters of the bloom filter on its own, and just tells
the kernel while setting up the bus how large it will be. This allows
flexibility to readjust the bloom filter metric or hash function without
changing the kernel.

> > To install matches for broadcast messages, use the KDBUS_CMD_ADD_MATCH
> > ioctl
> [...]
> > To match against other user messages add a KDBUS_ITEM_BLOOM item in
> > the match (see above).
> 
> Does this mean that eavesdropping (receiving unicast messages intended
> for a different recipient) is explicitly part of the kdbus API? This
> is

Note that the bloom stuff applies exclusively to broadcast
messages. Installing a bloom filter for unicast messages will yield no
results. This is because clients sending messages will not calculate and
attach the bloom filters for unicast messages, so there's nothing to
match again.

For eavesdropping we have a completely seperate connection
type. Connections to the bus of type "monitor" won't get a unique id,
and won't be able to send messages. However, they will get *all* unique
messages, and optionally the broadcast messages too, if they also
install a bloom filter.

Creating monitor connections requires privs in kdbus.

> fine if it's what you want, but I would caution against making it easy
> to do this by mistake - my recommendation would be to make sure that it
> can only happen if "eavesdrop" (or some similar jargon word) actually
> appears in your source code (see
> <https://bugs.freedesktop.org/show_bug.cgi?id=37890>).

Yeah, we should be very safe there. You cannot accidentally create
monitor connections.

> > The "memfd" concept is used for zero-copy data transfers (see
> > above). memfds are file descriptors to memory chunks of arbitrary
> > sizes.
> 
> If it works as well as it looks as though it should, then this seems
> like a great piece of technology on its own, even without the rest of
> kdbus.

Yes, absolutely. The graphics guys showed interest in using them for
passing pixmaps around, and in PA this would be the perfect choice for
the sample cache (when combined with volatile ranges, should we ever get
those on Linux, in particular)

> > With $UID replaced by the callers numer user ID, and $XDG_RUNTIME_DIR
> > following the XDG basedir spec.
> 
> When substituting XDG_RUNTIME_DIR into a D-Bus address, it must be
> escaped. (As an implementation detail, I think systemd only generates
> values for XDG_RUNTIME_DIR where the escaped version equals the
> unescaped version - but that isn't true in general, since it's a
> user-controlled environment variable.)

Our own "libsystemd-bus" does that already.

> > To additionally check against sender
> > names, use the KDBUS_ITEM_ID (for unique id matches) and
> > KDBUS_ITEM_NAME (for well-known name matches) item types.
> 
> Does this actually match how dbus match rules for well-known names work?
> I believe dbus match rules interpret a well-known sender/destination as
> meaning "must own this name or be in the queue for it", not "must own
> this name" as I assume kdbus does - but please check that, I could be
> mixing up match rule semantics with bus policy semantics. If I'm
> remembering this correctly, then you might need to add a
> KDBUS_ITEM_QUEUED_NAME in order to be able to implement AddMatch with
> compatible semantics.

What I wrote above was about NameOwnerChanged, which dbus-daemon only
generates when the real owner changes ownership. You will not get it for
queued names. The spec doesn't really clarify that, but I am pretty sure
that would be quite useless, if it sent them out just for queuing too...

Or am I misunderstanding you?

> > - The org.freedesktop.DBus "driver" service is not special on
> >   kdbus. It is a bus activated service like any other with its own
> >   unique name.
> 
> My instinct is that this is going to break applications in obscure ways.
> org.freedesktop.DBus is a strange mixture of a unique name and a
> well-known name (syntactically it's a well-known name, but it's reported
> to be its own owner like a unique name), because it's the thing that
> defines/implements well-known names.

This is actually a good point you raise. For the bus proxy I have been
forth and back over this, i.e. when we forward a message from the proxy
whether we should override the sender for it, and make sure the unique
name is replaced with the well-known name in it. I have changed this
around and back a couple of times. Currently we leave the unique id in,
and everything i tested works fine, including gdbus for its own internal
messages to the driver.

Note that all signals the driver traditionally sent out are now
synthetic, i.e. are generated in the proxy from a kernel message rather
than from the actual driver mini-service. When synthetic we already
write the well-known name into the sender field, not the unique id. This
means that only for method replies we still leave the unique id in (as
described above), but every implementation eats that just fine.

Overriding this for the method responses is trivial btw, as the metadata
of each message we sent actually includes a list of all well-known names
of the sender...

> If o.fd.DBus exists as a real name at the kernel level, I'd be tempted
> to implement it as a unique name with a known numeric value (either at
> compile time or at runtime), and have a special case in the kernel
> unique ID <-> D-Bus unique name mapping in libraries (at least, those
> with existing API guarantees like GDBus) - something like "unique ID 1
> maps to org.freedesktop.DBus, unique IDs 2+ map to :0.2, :0.3...", or
> perhaps "the special unique ID you were told by the HELLO ioctl maps to
> o.fd.DBus, and the rest map to :0.NNNN".

I don't think this is realy necessary, since we can trivially easy
override the sender in userspace for the method responses, and for the
synthetic signals we fill in whatever we like anyway (see above).

> Of course, if o.fd.DBus is just something that's synthesized by client
> libraries (like the special o.fd.DBus.Local pseudo-interface) in a kdbus
> world, then it's easy to give it backwards-compatible semantics, so
> please do :-)

Yupp, this is what we do.

> > - NameAcquired/NameLost is gone entirely on kdbus backends if
> >   libsystemd-bus is used.
> 
> This does make sense to me, though - it's easy for dbus and GDBus to
> emulate it from the HELLO ioctl (for the unique name),
> NameOwnerChanged(*, myself, *), and NameOwnerChanged(*, *, myself).
> However, NameAcquired and NameLost do have the special property that
> they are automatically received, with no explicit match rule needed.
> 
> To minimize breakage, please be careful to make sure that the kdbus port
> of any existing library (i.e. dbus and GDBus, but not necessarily
> libsystemd-bus) automatically subscribes to at least the subset of
> NameOwnerChanged signals that will allow it to synthesize NameAcquired
> and NameLost, and that it synthesizes these signals in the same order
> that would be expected from dbus (I don't immediately know whether that
> means "before NOC" or "after NOC" - the spec should say, but probably
> doesn't).

Yes, these ports need to make sure of this. Also note that the proxy
already synthesizes them for unique and well-known names and sends them
out even if you are not subscribed, thus compatibility is kept here.

Lennart

-- 
Lennart Poettering, Red Hat