DBus Extensions

[Thiago Macieira]

Ryan Lortie wrote:
>Application-visibility
>
>Having a type obtain a new possible value of NULL is merely a specific
>case of something that we should never do.  That thing is to change the
>meaning of the format for an existing client.
>
>Any application-visible changes need to be done in a backwards
>compatible way.  I believe that a useful way to think about this is with
>interfaces.
>
>Imagine we add a floating point type "f" to DBus.  As long as an
>application doesn't consume any signals or methods that have "f" as part
>of their interface then the application does not have a possibility of
>becoming confused.  This is a safe change in terms of
>application-visible changes.
>
>The only problem is that the application may consume "too many" signals
>by having match rules that are a little bit too lose.  One way to solve
>this might be to implicitly limit match rules to only return matches for
>messages that the client will understand.  I'm not sure what the
>negotiation mechanism between the client and the library might look like
>-- particularly in the case where the "client" is actually several
>possibly-differently-abled clients (ie: different libraries in one
>process).

What I think is missing is a "signature" field to the match rules. Most of 
the time, when you are adding a match rule for a signal, you know the 
signature you are expecting. This would alleviate one of the problems.

But there's still the issue of a program sending another new types just 
for fun. It would cause the remote program to close the connection, with 
the current implementation. That can be seen as a DoS vector.

One alternative is what you proposed: the bus automatically matches the 
message "version" against the client's protocol. This would prevent the 
problem you mention.

Another is for the library not to drop the connection if it receives a 
message it doesn't know how to handle. It should skip to the next 
message. It should only drop the connection if it can't make sense of the 
message at all.

I'd say doing both is a good thing. I'd like to avoid increasing the 
version number in the message too often, because it's only a single byte.

>*** We should only add new application-visible changes if they do not
>*** break applications that mind their own business by sticking to an
>*** old interface.
>
>..
>
>Wire format changes
>
>This is more difficult -- but it doesn't have to be too bad.
>
>I think that the most reasonable way to proceed here (excuse my bias) is
>for DBus and DValue to use the same serialisation format (( and I really
>like DValue's format :) )).

So DValue is going to use D-Bus's wire format? We can take some ideas from 
DValue, but we can't change the wire format now. Not substantially 
anyways. (If we were, I'd suggest using UTF-16 for strings, because it 
would make my life easier)

>The bus daemon (because it is currently the only implementation) is the
>easiest thing to update.  

It's not the only implementation. You can stop right there.

>As such, it should support any serialisation 
>format that we believe to be sane.  The idea here is to make things
>easier for the client libraries -- we want to keep the client library
>implementations simple.
>
>
>*** Make the server smarter in order to keep the client libraries
>*** simple.  It's easier to fix bugs in one piece of code.
>
>
>Each client (ie: connection) to the bus should have the following:
>
>  * a list of understood serialisation formats (with parameters, see
>                                                below)

The only serialisation format should be D-Bus.

>  * a preferred format (also parameterised)
>
>The bus should be prepared to accept messages from the client in any
>format that the bus understands.
>
>Upon forwarding a message to the client (from some other client) the bus
>should consult the destination client's list of understood serialisation
>formats.  If the message is already in a format that is on the list then
>the message should be sent.  If not, then it should be converted to the
>preferred format.  This mechanism also allows us to deal with
>endianness.

Uh, please, no. I don't like the idea of transforming a message 
on-the-fly. That would increase the delivery time and could introduce 
other problems.

Imagine the transformation were done for an "af" (array of 
single-precision floats) into an "ad" (array of double-precision floats). 
Sounds simple, right? Well, now imagine the array has a couple of million 
of items. Instead of simply copying bytes over, it would require an 
expensive operation, quite probably done using floating point 
instructions. And if it the original message were larger than half the 
max message limit, the transcribed message would be larger than the limit 
and could not be sent. 

You'd start seeing subtle glitches, when some messages are delivered to 
all clients, some are delivered to only a few.

So, for those reasons, I believe transcribing messages on the daemon is a 
bad idea.

I think we have to find a way to use the extension fields that exist 
already in the version 1 message.

>*** When adding new serialisation formats, we need not wait for everyone
>*** to update client libraries.
>
>A few words on how "worth it" it is:
>
>What's the point in having a "more efficient" format if the bus
>constantly has to do conversions?  There's still a point here, I think:
>the bus currently looks through the entire message anyway (for
>validation purposes) so it could just as easily do conversion at this
>point.

Sorry, I completely disagree. "Just as easily" sounds VERY wrong to me.

>In addition, the overhead of the bus doing this will always be 
>O(n).  Depending on the access patterns made by the client after
>receiving the message, things could be much worse there.
>
>The extra work on the server makes my life a bit nicer, and probably Rob
>Taylor's and Ori's too.  Having the messages come over a socket to us in
>DValue format is a gigantic win. 

Sure. Should we add QDataStream too? How will we communicate then?

I would like to add that QDataStream is used in a lot more applications 
than DValue, for the simple reason we've been using it in KDE since KDE 
2. If we just adopted it instead of DValue, we could talk to the KIO 
Slaves directly. And we probably could have maintained compatibility with 
DCOP too, because QDataStream was the basis of that protocol.

Good reasons, right?

Sorry, no. You're using D-Bus in the first place because you want 
interoperability. I agree completely that the nice ideas that DValue has 
can and should be added to the D-Bus message. What I disagree is dropping 
the current format completely in favour of a new one. 

At least, at this point in time, without exhausting other possibilities.

>The fast lookups and other nice 
>formats of the DValue serialisation format will also make other people's
>lives easier.

Adding an index to the extension fields in the D-Bus message is easy too.

>Before we proceed, though, it really makes sense that we try to get a
>consensus on this format.  The normal case of bus operation should not
>be that every message gets reserialised by the daemon.  Our goal should
>be, at any given time, to have a certain message format that we actively
>encourage implementers to use.  Ideally, most of the time, the
>conversion facilities would cover the unusual cases of old client
>libraries and mismatched endians.

Mismatched endianness is not a problem. The library can decode that 
easily.

If you want a consensus, you're not going to get it.

>*** We should only add new serialisation formats if we believe that they
>*** have hope of being widely adopted.

No new argument here.

>Of course, new extensions to any serialisation format could be made very
>easily, assuming that they satisfy the requirements above (ie: adding
>new types is fine so long as old applications are not broken).
>
>Implicit in this is that any additions only serve to increase the number
>of byte sequences that represent valid messages.  All formerly-valid
>byte sequences continue to correspond to exactly the values that they
>did before.
>
>The negotiation of which "new types" are supported forms the
>parameterisation that I mentioned above.  You tell the bus:
>
>"I support original dbus serialisation." (actually, this would be the
>implicit case if you do no special negotiation at all)
>
>or
>
>"I support DValue serialisation, plus chocolate-sending capabilities."

If you want to use DValue, send an "ay" (array of bytes) opaque blob. 
Don't expect others to understand that.

That's what my conceived-but-not-implemented DCOP bridge would do for 
complex messages.

>If you merely say "I support DValue serialisation" and then somebody
>sends you chocolate then you lose.
>
>The semantics of "you lose" go something like this (but basically mean
>that the client will never-ever be sent chocolate):
>
>1) If someone attempts to send you chocolate as part of a method call
>then you hear nothing and the sender gets a "server is pwned" error.
>
>2) If someone attempts to send you chocolate as part of a method call
>reply then you receive "server is pwned" error.
>
>3) If someone attempts to send you a chocolate-carrying signal then you
>just don't get it at all.

2 and 3 are unacceptable. I could live with 1.

>The reason for this is that if you send chocolate to someone who is
>incapable of digesting it (libdbus-dog?) then you'll have quite a mess
>indeed.

:-)

>*** New types should only be added with explicit feature negotiation and
>*** the bus should only send them to clients that have explicitly
>*** requested them.

Agreed.

I have had ideas about a "bus of busses" which would be an extension of 
the addressing scheme. The basic idea was what you said above: if you 
don't request it, you won't even notice it is there.

>Where should validation of message contents be done?
>
>Is it appropriate for an application to overflow some buffer based on
>"evil bytes" sent to it by a trusted bus daemon?

Eh? No. Overflow is never appropriate.

>Is it appropriate for it to fail an assert()?  Throw an exception?

It should never assert, crash or cause the program to terminate if it 
read "garbage" from the network.

>Must the error be caught at the time the message is received or might it
>be caught later on while deconstructing the data object?

When it is received. If it's done at deconstruction time, the problem 
propagates to the binding and possibly user code.

>If the standard is that client should always validate bytes sent to them
>by the daemon then why is the daemon also performing validation while
>forwarding?

libdbus-1 is not the only implementation. It's useful for developers 
writing new implementations to know that the message they have received 
is valid, so they must be able to parse it. In other words, if they fail 
to parse it, it means there's a bug in the code, rather than it was 
garbage that was received.

But the bus validation could be disabled on the session bus for speed 
improvement.

-- 
  Thiago Macieira  -  thiago (AT) macieira.info - thiago (AT) kde.org
    PGP/GPG: 0x6EF45358; fingerprint:
    E067 918B B660 DBD1 105C  966C 33F5 F005 6EF4 5358
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
Url : http://lists.freedesktop.org/archives/dbus/attachments/20070811/1ae19353/attachment.pgp