C convenience lib (was Re: Is dbus_pending_call_set_notify() thread safe?)

[Havoc Pennington]

Hi,

Thiago Macieira wrote:
> Nothing. I've just realised that there's a check to make sure it's no-op. 
> I hadn't seen it before, which made me think that calling it more than 
> once was dangerous.

I think you were right about this at some point in the past btw, it just 
got fixed at some point and the de facto situation is everyone just 
calls dbus_threads_init_default(). In theory if nothing in your app 
needs threads, they are never initialized and libdbus is mildly more 
efficient. (Probably saving 5% overhead or something.)

> So, if you're writing a mainloop-specific code, you must assure that the 
> binding is loaded into memory and is working. If you know that is the 
> case, then why not use the binding already?
> 
> Therefore, I argue that the only valid reason for using libdbus-1/wrappers 
> directly, instead of a binding, is to write mainloop-agnostic code. And 
> since you must not assume anything about mainloops, you must handle your 
> connection's dispatching yourself. That means dbus_bus_get_private and a 
> private thread for handling asynchronous events.

Hmm. OK let me break down some multiple issues again ;-)

1)

So I think it would be OK for a library or plugin to assume "something 
will dispatch the session bus connection" and that would work fine, with 
no direct dependency on say dbus-glib. All a lib/plugin really has to 
know is that something will be doing the read/write/dispatch task.

Contradicting my previous mail a bit, conceivably this allows some 
simple plugins or libraries to avoid caring about the main loop, though 
it would only work if the plugin never needed a main loop for any 
_other_ dbus-unrelated reason.


2)

Something of a special case is when a library or plugin is a dbus object 
system mapping.

There's some split I think between a mainloop/dispatcher, and an object 
system mapping.

For example, in the Mugshot client app, we use a lame object system 
mapping I came up with ad hoc. However, we use the mainloop/dispatcher 
from dbus-glib, and we have libraries in the same process which I think 
use dbus-glib's object system mapping.

As another example, you can use the glib mainloop/dispatcher with the 
Python object system mapping, for example in a PyGTK application. [1]

The object system mapping just assumes "something will do 
read/write/dispatch" and that's all it has to know.

This dispatcher vs object system mapping distinction does not change 
your fundamental point, but it does mean the thing that has to be 
globally decided on a shared connection is smaller and less consequential.

It sounds like this doesn't work in QtDBus since the Qt object system 
mapping assumes the Qt dispatcher is used, but AFAIK the GLib and Python 
bindings for example don't make assumptions about the dispatcher (other 
than "it exists")

Anyway, to simplify slightly (since there are actually N shared 
connections[2]), the question here is whether the conceptual global 
variable:

  DBusConnection *the_session_bus;

is in libdbus or in dbus-glib and QtDBus.

I would say that either way, conceptually this global variable "belongs" 
to the single dispatcher in use for the given app.

There's some very small utility to reducing the code each dispatcher has 
to write slightly, by having libdbus deal with the global variable, 
thread locking it, and zero'ing it out on disconnect.

I think there's some utility to allowing say Python object system 
mapping and GObject object system mapping to share a single connection 
and single dispatcher.

If they don't share that, I don't think it's a tragedy; an object system 
mapping that requires a specific dispatcher can use a private connection 
and the only harm is a bit more overhead.

(Though I think people do underestimate how big DBusConnection is and 
the potential network overhead of creating one, esp. if we make TCP work 
well.)

3)

The shared connections are intended to be "claimed" by some dispatcher.

So it is fair game, IMO, for QtDBus to entirely take over the shared 
connections and just say "if you want to be in-process with QtDBus you 
have to use the shared connections as we set them up."

It sounds like a lib or plugin as I described in 1) above won't work 
anyway, but QtDBus can still own the shared connections if it likes.
It would just be a policy of QtDBus that it forbids non-QtDBus usage of 
shared connections.

That is effectively the policy anyway, since QtDBus is not dispatching 
the shared connections, a plugin or lib as described in 1) won't work 
anyhow. Given that this is already true in effect, there's no reason 
QtDBus can't just take over the shared connection. There may be no 
benefit to doing so, but there's also no reason not to.

We could add some API to essentially "assert" that only one dispatcher 
claims the shared connections. That might serve some of the same 
preventing-weird-bugs purpose as deprecating shared connections.

dbus_set_shared_connection_dispatcher(const char *name,
                                       gboolean allow_other_dispatcher,
                                       DBusConnectionSetupFunc func,
                                       void *data);
or something like that.

If allow_other_dispatcher it means we don't care if someone else 
dispatches as long as they call our registered callbacks (so e.g. a 
plugin could set a dispatcher "just in case", like threads_init).

However QtDBus would have to do allow_other_dispatcher = false which 
would mean we abort() if someone else had already taken over the shared 
connections.

We could also have something like

  const char* get_shared_connection_dispatcher()

which would allow object system mappings to do an
assert(I have the right dispatcher)

4)

The DBusConnection API _is_ designed for multiple object system mappings 
to share the connection; that's the purpose of the "object tree" for 
registering what object paths you want to handle, and the return values 
from the callbacks like HANDLED vs. NOT_YET_HANDLED.

This has worked fine for me for object mappings like the dbus-glib 
GObject one combined with my lame custom one, a combination I've used 
successfully.

I do think it's a bug if it's impossible to coordinate multiple object 
system mappings on one connection using the available APIs, though if we 
deprecated shared connections it is obviously not a very important bug.

I'm tangentially interested in why QtDBus wants to own the message queue 
and object tree rather than coordinate, if the reasons are likely to 
apply to other libs or object system mappings using the connection.

5)

If we deprecated shared connections we could actually remove them 
internally and always create new connections, since the shared getters 
do return a reference count and make no real guarantees about sharing, 
the API contract is just that the connection *could* be shared.

However, legitimate code people are using now could end up creating a 
heck of a lot of new connections if we did this. It would work but be 
crazy inefficient.

6)

The reason to deprecate as you say is "to avoid silly errors of 
applications using the connections shared by libdbus-1"

Do we really have a problem here in practice, though? I would expect 
that if someone did this their lib or plugin or whatever would just not 
work and they'd notice pretty quick.

If this is biting people I'd like to address it somehow.

7)

This whole conversation may be relevant to people who are wanting to 
have the binding own the message queue and the marshaling.

i.e. there was this discussion at GUADEC (see my notes from there) about 
offering an API that just created the socket and did the auth protocol, 
then returned a file descriptor.

Then allow bindings to parse and build messages, avoiding the 
queue/object-tree in DBusConnection and avoiding the DBusMessage object.

Obviously this means that bindings would have to be both dispatcher and 
object system mapping, and that shared connections would be unworkable.

If you think about it, basically the whole purpose of DBusConnection is 
to share the connection - if we didn't want to share the connection, we 
could just have a "connect and auth, return fd" and then possibly also 
marshal/unmarshal helper API.

There are a couple other small DBusConnection purposes (e.g. in theory 
we could have a non-descriptor-based transport like an X client message 
tunnel), but they aren't too important. The main point of DBusConnection 
is to coordinate multiple users of the connection socket.

This "just give me an fd" API is kind of hypothetical for now though 
unless I or someone gets time to hack on it ;-)

8)

As I also said in that GUADEC mail, for C apps, I think apps are 
suffering most from lack of high-level API to do things like 
introspection, single-instance, tracking bus names, and matching 
signals, so we probably should think about that in addition to tweaking 
libdbus to death and/or reimplementing parts of it...

Havoc

[1]
(Though I think dbus-python didn't use the shared connection, I thought 
the reason was something to do with refcount bugs, I'm not sure.)

[2] the shared bus connections (dbus_bus_get()) are essentially just 
global variables and the implementation of dbus_bus_get() would not be 
especially hard to redo in a binding. However, there's also a 
not-really-ever-used-right-now-but-maybe-someday-useful-possibly sharing 
of connections by server GUID, which has the theoretical purpose that if 
you did a CORBA-looking usage of dbus, with "IORs" and no bus daemon, 
you would want to avoid creating a new connection object for every 
method call. Thus there's basically a hash from the server GUID (which 
would be in the "IOR") to shared connections. Anyway, we can ignore this 
case, but it does exist in the code for historical reasons.