D-Bus optimizations

[Mikkel Kamstrup Erlandsen]

On 03/07/2012 07:10 PM, Rodrigo Moya wrote:
> On Mon, 2012-02-27 at 13:04 +0100, Rodrigo Moya wrote:
>> Hi
>>
>> Just published a blog entry[1] about some recent work we have been doing
>> at Collabora to optimize D-Bus. So could you please have a look and
>> comment on it?
>>
>> cheers
>>
>> [1] http://blogs.gnome.org/rodrigo/2012/02/27/d-bus-optimizations/
>>
> after some (failed) discussion on the linux-netdev mailing list, here's
> a 2nd blog post about the situation:
>
> http://blogs.gnome.org/rodrigo/2012/03/07/d-bus-optimizations-ii/
>
> so, could you please have a look at it and comment?
>

Hey Rodrigo,

I've been thinking quite a lot about this topic for a while now and I am 
happy that someone has taken it up. First some background.

We're sending a *lot* of data over the bus in Unity. And by "lot" I mean 
truck loads. So, naturally DBus performance is of great interest to me.

However - we are not seeing DBus being the bottleneck for performance. 
Quite far from it in fact. This is probably because all protocols for 
the Unity infrastructure has been aggressively optimized wrt to the 
number of roundtrips (sacrificing message simplicity and size). Even to 
the point some would call hacky - I call it clever ;-)

The way we've achieved this is by regarding every DBus roundtrip pretty 
much as a HTTP request (you really don't wanna be sending lots of those 
out). And by regarding DBus as a framework to build a *protocol* and not 
as a framework to build an object oriented remote procedure call layer. 
Granted - this is against DBus tradition and documentation; but this is 
something that I think we need to change.

I am seeing time and time again people mapping OO designs to bus objects 
and it can only lead to performance problems. They are doing this 
because this is what all online examples tell them to do. One must think 
in batched calls, stateless requests, and idempotency to devise fast and 
robust protocols. I am sure we could come up with a simple cheat sheat 
to guide people through this. "You have a method that returns a single 
boolean? Unless you have really *nothing* else interesting to return 
you're doing it wrong" ;-)

One point about DBus which I'd very much like to preserve is that 
there's no magic to it. It's really just an old user space daemon 
written with the common tools we all know. This is a great asset that we 
should not give up easily.

Keeping these things in mind here's what I believe would provide for the 
best approach:

  * Improve the documentation with a suite of best- and worst- practice 
examples.

  * Stress in the documentation (also with an example) that it is easy 
to set up a "private sub bus" using p2p DBus and that this shaves off 
roundtrips because the bus host lives inside your own process.

  * Look into implementing some sort of "fast side channel negotiation" 
like Havoc proposed. Keeping it disjoint but nicely integrated with 
DBus. If there was a good, super fast, and well documented solution for 
this I am quite sure that it would become a de facto standard very rapidly

  * Determine precisely why the daemon gets choked up. And figure out if 
some of the more down to earth solutions can provide the needed boost. Fx:
   - Can the daemon or client libs possibly apply some clever pipelining 
of writes/reads to minimize the number of ctx switches?
   - Optimize message parsing/building?
   - Would batched Add/RemoveMatches(in as match_rules) calls improve 
anything (that is, if added to the spec)?
   - Would threading in the daemon help out? Either just some simple 
producer/consumer thing, or a more elaborate work-stealing-threads kinda 
thing[1]?


  * Also, as others have asked:
  - What use cases are we optimizing for? Is it latency, throughput, 
both, or something else?
  - What is the performance target? Personally I wouldn't see much gain 
in a 2-4 times reduction in roundtrip latency. We'd still have all the 
same problems. We'd have to be talking orders of magnitude to really get 
me jumping out of the chair :-)


Cheers,
Mikkel

[1]: Forgive a total shot in the dark: I think I recall Lennart adding 
some work stealing magic to Pulse Audio. Perhaps something can be 
shared/lifted?