using dbus for higher-volume messages?

[Havoc Pennington]

Hi,

On Sun, 2005-11-27 at 17:11 +0100, Johannes Berg wrote:
> Been thinking a while about some kind of local IO mechanism for d-bus
> that doesn't need to copy all the data around all the time, for
> high-volume applications like scanning. Now, maybe d-bus isn't the right
> tool for that kind of job, but it was more of a thought-experiment to
> see if it's at all possible.
> 
> What I've come up with is visible at
> http://johannes.sipsolutions.net/Projects/dbus_zerocopy
> 
> If anyone thinks this is worth it at all I may take a closer look at the
> d-bus sources. But feel free to shoot me down with "This isn't want
> d-bus is intended for" :)
> 

Thanks for the write-up. See also some archived discussion about
how/whether to pass file descriptors over dbus; there are some
complexities iirc, esp. with the current codebase.

One thing that might be worth exploring is just a simple API _only_ for
passing shared memory blocks, that would use a separate socket. So the
idea would be to use dbus for rendezvous but shove the data over this
other API. One idea might be that each shared memory block has a unique
string ID, which is passed over dbus as method call arguments. The
shared memory block API then looks something like:

  // returns the unique ID of the data block
  char* send_data(const unsigned char *block, size_t len);

  boolean get_data(const char *block_id, const unsigned char **block, size_t *len, Error *error_return);

One way to do the "separate socket" thing is relatively transparently;
introduce a dbus type for the shared memory block, but on the dbus
socket its value is just the unique ID, and the block itself is pushed
over an additional socket.

Another way is to make the app developer deal with it, so there's a
separate API for the data block stuff and you just stuff the unique ID's
into a dbus string.

Rather than a persistent separate socket for this, one approach would be
that you make up a new socket name each time (you want a randomized
socket name for security anyway) and the unique ID is simply the socket
name.

I think a relatively heavy "setup cost" is probably OK for this
application right, since the point of this is to transfer huge pieces of
data.

Another thought, you have:
dbus_message_iter_get_memory

I wonder if it would be better to mark something "shared mem" on the
whole-message level. i.e. maybe the shared memory thing could be a
special kind of message, instead of a special kind of message argument.
Don't know, it's just a thought.

I'm just throwing out some ideas - obviously I haven't thought about it
all that much.

Havoc