Parallel method calls to server

[Havoc Pennington]

Hi,

Pallavi Rao wrote:
> In a serialized/synchronous method call, two different
> clients, client1 and client2 call the same method A on
> a serving object and they are worked upon one after
> the other. So that after the first method call from
> client1 is done with, the second call from client2 is
> taken care of(and replies are received in the same
> order at the clients).

Clarifying one point, the ordering is not guaranteed between two 
messages sent by two different clients - ordering is only guaranteed 
with respect to each client.

For each message queue (DBusConnection), that queue is ordered. However, 
if there are two queues, there are no ordering guarantees between them.

So if you send two messages from the same DBusConnection, the server 
will receive and process those messages in the same order you sent them. 
But if you send one message from connection A and another message from 
connection B, the server may receive and process those messages in any 
order.

Obviously you can impose stronger ordering guarantees by doing your own 
synchronization. In other words, if you send a message on connection A, 
wait for a reply from the server confirming it was processed, and only 
then send a message on connection B, then you know the first message was 
processed first.

But if you send both messages at the same time, on two different 
connections, without synchronizing by waiting for the reply on the first 
one, there's no ordering.

This is a little confusing since there are three processes involved in a 
typical method call, let's say the method caller, the bus daemon, and 
the method handler.

In the typical case, the method caller and the method handler each have 
one DBusConnection to the bus daemon. The bus daemon has a bunch of 
DBusConnection, and it moves messages from each connection to other 
interested connections.

If the method caller sends three method calls, those are all going 
through the same connection; the bus daemon will thus read and process 
them in the same order they were sent. The bus daemon will then write 
them to other connections, including the message handler's connection, 
in the order of processing. The message handler will receive the three 
method calls, in order, on its one connection. The message handler then 
sends back three replies, which stay *in order the handler sent them* as 
the daemon processes them, and are received back by the message caller 
still *in order the handler sent them*.

So nothing gets reordered by libdbus or the bus daemon, as long as it's 
on a single connection. However note that the method handler *can* 
process each method call out-of-order. For example, if the method 
handler is multithreaded and uses a different thread for handling each 
message, you would expect replies to be out-of-order.

Also, again if there's a second method caller B, and it also sends three 
calls; there is NO ordering with respect to the messages from the first 
method caller A. The method handler could process three methods from A, 
then three from B; or it could do B first then A; or it could interleave 
those methods.

In short:

  - each DBusConnection is a queue, and things in a queue stay in order.

  - after a method handler removes messages from a queue, it has the 
option to process them out-of-order (for some messages this makes sense 
and for others it would be a disaster - it's a domain-specific question)

  - when the bus daemon forwards a message from an incoming queue to an 
outgoing queue, it preserves the order of the incoming messages.

  - the bus daemon reads from all its incoming queues and writes to its 
outgoing in arbitrary order, so there is no ordering between two 
messages unless they are coming from and going into the same queue

> In a parallel/asynchronous call, a thread(?) is
> generated per each single call to this method A. So
> for two calls from two clients to a single method at
> the same time, two separate threads run in parallel to
> finish work at the same or almost the same time.

This is not exactly the case; libdbus never creates threads, and 
dbus-daemon is not multithreaded. However, dbus-daemon *is* completely 
asynchronous (it never blocks), which means that it behaves exactly as 
if it had a thread per connection. So you can think of it as 
multithreaded if you like and you will draw the right conclusions.

You would draw the *wrong* conclusions if you think of the bus daemon as 
having a thread per *message*.  The bus daemon processes only one 
message at a time, in order to preserve the ordering guarantees for 
messages from the same queue. The bus guarantees that if it reads 
message A1 then A2 from the same connection, it will write those 
messages to outgoing connections in the order A1 then A2.

If the daemon had a thread per *message* then it might reorder stuff in 
all kinds of ways, unless it had a big global lock (in which case the 
threads would have no purpose). So multithreading the bus daemon is 
almost certainly pointless because the protocol guarantees a fair bit of 
synchronization.

In your own application, you need not provide the same ordering 
guarantees as the bus daemon. Once you read messages from the ordered 
incoming queue (DBusConnection), you can process them out-of-order if 
you like, using either threads or nonblocking IO.

There's no global answer to whether this makes sense. For example if you 
have two method calls like:

  Init();
  DoStuff();

then you probably don't want to process those in two threads, since the 
Init() needs to happen first and then the DoStuff().

But if you have two unrelated method calls, the order in which you 
process them might not make any difference and parallelization could be 
a speed win.

The method caller can always force two methods to run in sequence by 
waiting for a reply to the first before sending the second.

> In
> this case a reply from a thread is to be sent back to
> the right client to which it belongs.
> Does the dbus directly deal with the threads in this
> case, or should that all be done inside the server
> object? (I guess dbus has generally not anything to do
> with this, but I am not sure.)

libdbus won't ever create threads, that's up to the application.

> In other words, I am searching for information/
> documentation on how the multithreaded applications
> are dealt on the dbus??

There is a lock on each DBusConnection, so it is safe to send messages 
(method replies or method calls) from multiple threads. For example, two 
different threads might each dbus_connection_send_with_reply_and_block() 
at the same time. Or if you are handling two messages in parallel in two 
threads, each thread can safely send back its reply to the same 
DBusConnection object.

However, it is the application's problem to figure out when parallel 
processing and reduced ordering guarantees are in fact acceptable. For a 
method caller, usually you can just call methods willy-nilly from any 
thread you want, since presumably each method caller thread is unrelated 
to the others.

For method handlers, if you have any ordering issues between your 
methods (such as an Init() call that must be handled first), it's 
probably a bad idea to run them in parallel, unless you provide for some 
kind of locking that preserves the important ordering.

But as long as your method handler is stateless (like an HTTP server), 
handling multiple calls in parallel should not be a problem.

You should almost certainly avoid calling dbus_connection_dispatch() (or 
other functions that cause a dispatch, see the docs) from more than one 
thread; because the dispatch will happen in whichever thread you call 
dispatch() from, and this is unlikely to be sane.

> Is it also possible to have new and proper processes
> with unique PID's instead of threads, per each call in
> the parallel implementation of serving applications.
> (Then the processes, for example, can have unique
> connection names !)

It sounds possible (in effect the bus daemon is doing something like 
that), but you'd have to design and implement such a system yourself. 
libdbus is just giving you the stream of incoming messages in order; 
your application is responsible for any parallelization, using threads 
or processes or any other mechanism you like.

Havoc