Starting the kdbus discussions

[Simon McVittie]

On 17/01/14 19:21, David Zeuthen wrote:
> On Fri, Jan 17, 2014 at 10:06 AM, Lennart Poettering <mzqohf at 0pointer.de> wrote:
>> I am pretty sure that the dbus thread gdbus uses is a poor design
>> choice.
> 
> The reason for GDBus using a
> worker-thread has to do with ease-of-use in both multi-threaded
> event-driven applications (e.g. using a main-loop mechanism and async,
> callback-based facilities to send/receive D-Bus messages and
> worker-threads using sync D-Bus facilities to send/receive) as well as
> command-line programs where you don't have a main-loop at all.

Expanding on this:

The three things that need to happen are:

* If a thread is blocking on a synchronous D-Bus call
  (bad idea for a UI/network/service thread, but OK
  for a worker thread or a simple command-line tool),
  it receives the reply when appropriate

* If an async message comes in (either a reply to an async
  call or a signal), its callback is called in the thread that
  expected to receive it, and not in any other thread[1]

* Someone does the actual I/O

and in stream-based D-Bus (but perhaps not kdbus, depending how your
syscall API looks), there's an additional requirement:

* Exactly one thread does the actual I/O, so that threads don't
  read partial messages and get confused about where the message
  boundaries fall

libdbus tries to do this while share one D-Bus connection between
arbitrarily many threads, and arrange for it to work like this:

* if a thread has taken responsibility for "dispatching" the D-Bus
  connection (typically the one with the mainloop), that one does the
  actual socket IPC;

* or, if not, but at least one thread is blocking on a synchronous
  D-Bus call, then exactly one of them does the actual IPC

* async callbacks are called in the "dispatching" thread, or if
  nobody is "dispatching", build up indefinitely in memory

This turns out to lead to a confusing architectural train-wreck, and
still doesn't get callbacks called in the right main-context - they all
turn up in whatever main context is "dispatching", and if you want them
to go elsewhere you have to post events between threads.

I'm not sure how QtDBus (which wraps libdbus) deals with this: I think
it might do it by asserting the policy "async callbacks always go off in
the Qt main thread, which is where I dispatch the connection; you must
run the Qt main-loop in your main thread", and in practice that's how
people use Qt anyway, so it works out OK?

Anyway, GDBus' D-Bus worker thread avoids this, by guaranteeing that we
know where the actual IPC is taking place - that worker thread. GDBus
uses GLib's GMainContext machinery to do the necessary event-posting
between threads to get the callbacks called in the right main-context
(approximately "the right thread") - and because this *always* happens,
even for the common and heavily-tested case of "only one API-user
thread", we know it actually works (contrast with libdbus, which wasn't
actually thread-safe for several years despite claiming it was).

If kdbus' sysctl interface is atomic with respect to sending and
receiving messages, then you've solved the problem of sending messages
and having them not interleave, but not the problem of getting incoming
messages to the main-context that was prepared to deal with them.

    S

[1] in GLib, strictly speaking this should say "the main-context that
    expected to receive it", since main-contexts aren't tied 1:1
    to threads; in Qt, each thread implicitly has exactly one
    thing-equivalent-to-a-GMainContext