My comments on D-BUS

[Havoc Pennington]

On Thu, 2005-01-20 at 21:40 -0500, David A. Wheeler wrote:
> * I didn't see any reflection capability. 

It's undocumented but exists. You call Introspect() on an object path
and it returns a big XML string with the object instances at or below
that path along with the method information.

Implementing Introspect() is currently left to the bindings though so an
object implemented in raw libdbus will frequently suck it up and not
have the method.

The choice of xml is because bindings can trivially create and parse
that, all languages have a few hundred xml parsers available. The other
reasonable choice would be to use dbus marshaling format, but I think
that has some disadvantages (among them, you can't save it to a file as
a nice human-readable IDL).

> * If you're going to make this a standard way for interoperation,
>    consider eventually submitting the spec to a standards body.
>    The IETF might be a plausible place.  You'll have to be able
>    to justify why it's worthwhile, a point I'll get to in
>    a later email.

Cross that bridge when we get there... I think making it work and be
useful is the first step.
I don't have a lot of time available.

> * In the glib mapping (for spec and the tutorial),
>    s/dbus_g_proxy_end_call()/dbus_g_proxy_make_call()/
>    or some other name.  "ending" a call makes it sound like
>    you're cancelling a partly-completed call, but really,
>    you're finishing up the call, having it sent, and waiting
>    for a reply if you're waiting.

Normally this isn't for having it sent, it should have been sent in the
main loop. So you're collecting the reply. It will block *if required*
to either write out the call or read the reply but the "normal" async
mode would be to only call end_call() when you get a callback that the
reply has arrived.

wait_for_and_collect_reply() or something might be better, though that
particular example seems long.

> * Why is the reply optional if one isn't expected?
>    It seems that a reply is: (1) required or (2) shouldn't happen.
>    If you want a "reply optional", I'd also like to see a
>    "do not reply" markings... for many programs, I'd expect
>    that if they don't expect a reply, they don't want to have
>    to program in handling a "possible" reply.. they just don't want one!
>    And it'd be clearer if caller could clearly state
>    "don't send me a reply, I will NOT use it".

I would say there's no way for a reasonably-programmed caller to get
confused by an unexpected reply, because if it wasn't expecting a reply
it normally wouldn't have even saved the serial cookie to be able to
identify the reply. Remember that the target use case is desktop apps
where each app has a pile of libraries and all those libs share a dbus
connection, ignoring each other's messages. This is exactly how xlib
works.

However I think it's reasonable to write a callee that doesn't bother to
special case the "no reply expected" flag and skip the reply.

The purpose of this flag is purely optimization.

> * Before you set this in stone, I think you NEED to figure
>    out how to handle sending things over a wider internet.
>    People don't want to have to deal with 15 different RPC
>    mechanisms

People are kind of screwed then, there are already at least 1 million
afaict. ;-)

> , and I fear that it may have to have fundamental
>    changes to handle internet-level commo.  Maybe not; the
>    byte-ordering marker is a clear help, for example... but
>    until someone's thought it through, it'll be a big question.

I don't really know how to figure out if fundamental changes are
required, because I don't know what the use cases would be. I'm very
uncomfortable trying to architect around hypothetical apps, and I know
that IPC systems with the internet in mind are either substantially more
complex than dbus, or really simple because they are just http with a
pretty bow.

> * There should be UTF-16 and UTF-32 value support.

There are some real problems introduced here, pretty well-covered by the
thread sometime in the last month about implicit type conversion. If you
have multiple string formats, you either have to implicitly convert
between them (quite a few problems with that) or you have to treat them
as genuinely distinct types. If they are genuinely distinct types, then
if I have a method that takes a String which encoding should I use? All
the java programmers will use utf16 and all the unix programmers will
use utf8 and it will be a nightmare and a half.

So the only way I really know to do this would be to make a single
STRING type with an encoding tag at the front of it, and always return
strings as a copy in case they need to be recoded into a larger buffer.

Java is going to have to copy the string anyhow into a native string
type... and UTF-8 is pretty much the standard for both of the defined
use-cases (systemwide unixish systems, and per-session unixish
desktops). If you do implicit conversion that means adding a copy inside
libdbus, so java would get that copy and then copy it again.

Anyhow, it seems like a big can of worms and my leaning is to leave it
closed... human-readable strings are going to be shortish (a whole book
is going to transfer and recode pretty quickly) and machine-readable
strings are going to be XML which is already tons of parsing overhead.

After my last round of changes, the data validation is so slow that
string recoding isn't even going to appear in the profile ;-) though I
suppose I would like to fix that.

> * There should be a 16-bit signed and unsigned value
>    (this was in, and then removed in December).  This is
>    needed to simplify the creation of bridges to other
>    RPC mechanisms (CORBA, for example, supports both),
>    and to support arrays of these values.

I'm not sure this is worth optimizing, since the frequency of "short"
vs. "int" is incredibly low in most code. There is overhead and
complexity to each additional type, they make dbus harder to use.

I can see that it might avoid some hacks for bridges to existing code,
so maybe it's worthwhile.

dbus the wire format never supported 16-bit, the removal in december was
just a one-line diff to drop the uint16 typedef (it was unused)

> * Keep type 'Dict'. Dictionaries are _NOT_ the same as
>    'an array of (String, Value)'; arrays have an ordering,
>    and it's perfectly reasonable for a receiver to
>    expect that the ordering has meaning.  Many languages (Python,
>    Perl, etc.) have a built-in dictionary type where the ordering
>    of the strings is irrelevant.  Having a way to
>    convey 'order doesn't matter' seems useful, so that their
>    bindings can immediately map such information to the
>    built-in dictionary, without worrying about preserving
>    an order when it's irrelevant.

Sure, but what you need for this is simply a flag that the array of
(string, value) has no duplicate strings and has irrelevant order. The
wire format is going to be array of (string, value), there's no other
reasonable format. So this flag will either be in the introspection data
or in the signature somehow.

> * Are strings allowed to embed a NUL byte, since their
>    length is encoded?

Nope, strings are human-readable, nul isn't. The length is just there so
strings can be quickly skipped over without scanning through them.

>   Or, if you want to do that, should
>    you be required to send an ARRAY of BYTEs?

Exactly.

> * In the "Valid names" section, you often say
>    "[A-Z][a-z][0-9]_" and so on.  That's confusing; in normal
>    regular expressions that's a phrase with exactly 4 characters
>    always ending in underscore.  Why not just say:
>    characters matching the regular expression "[A-Za-z0-9_]"?

No reason, didn't think about it much. The "spec" is not exactly
precise.

> * In the "Valid names" subsections, regular expressions
>    that embody all the different rules as well as the specifics
>    below.  Regular expressions are unambiguous, shorter,
>    and I think many understand them better anyway.
>    E.g., in "interface names", say
>    "that apply to interface names;
>     such names must completely match the regular expression:
>        [A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*)+
>      Note that specifically:..."

of course, regexps themselves aren't all that standardized ;-) though I
guess you have nothing exotic in that one.

When coding from a spec I think the "check this, this, this" approach
can be easier; though the risk is something like the POSIX spec
for /bin/sh command line parsing, where there's no possible way to
specify it except by writing down whatever algorithm the original
implementation used, since there's no sane grammar to it.
 
> * Interface names - must be 3 bytes in length, not just one,
>    since 'a.b' is the shortest allowed.

True

> * In Signal emission - can you include parameters like a method call?
>    Why not?  Many languages that originally limited exceptions to
>    strings (like Python) have switched to permitting more information.

Yes, signals can have arguments as with any message.

> * There need to be a bunch of standard signals used for errors, e.g.,
>    wrong type, no such method, etc.  Consider including more info
>    to help debugging.

There's a special error type rather than signals, and there are some
standard error names and errors standardly have a string as first arg.

> * In the long run, at least a 'suggested' IDL would be useful.
>    Is there a serious problem with using CORBA's IDL, possibly
>    modified, even if you don't use its generator etc.?

No problem with it, no. I was going to use the same XML format returned
by Introspect() probably. But I like how DCOP/moc work, where writing
IDL isn't required. With introspectable languages it also isn't
required. IDL certainly contributes to CORBA clunk.

> * If you do change the protocol so that argument signatures
>    are sent as a separate string (instead of in-band with data),

right, the signature is now separate and the version is now 1.

> * If you change the protocol so that argument signatures are
>    sent as a separate string, you might want to change some of
>    the bindings to exploit that (to implement some type-checking
>    even when you're building the call dynamically). E.G.,
>      whatever_begin_call(..., typestring)
>        now, whenever you add a parameter, compare to see if the
>        value is what's expected... this even lets you auto-promote
>        if you want to (e.g., convert ints to the "correct" value).
>    There are pluses and minuses to doing this, but it might
>    be worth thinking about.

Yeah, I think there are a large number of improvements and features
possible in bindings (that I am conveniently ignoring by punting them
all to bindings...)

> * You ought to at least note that for some purposes you might
>    want to use another RPC technology, and then bridge it to D-BUS.
>    There are so many others (CORBA, DCE, SOAP, ...) that should
>    at least be acknowledged.  Ideally, you should describe HOW
>    at least some of their concepts match.

These are conceptually just another binding and the way it works is the
same way the language bindings work. dbus started out a bit more weird
but these days it maps very closely to the sort of "normal"
objects/interfaces/signals model that Java, C#, GTK+, Qt, etc. all use.

My experience with IPC bridges is that IPC is unreliable and difficult
enough without having two semantically incompatible systems involved ;-)
so I'd be awfully tempted to document "how to bridge" by saying "try to
avoid it, and if you can't then good luck, you are braver than I"


Anyhow... for the directions you are thinking, some work that comes to
mind is converting the spec to be more formal and precise; or coding on
a more "black box" test suite (the harness for that is already in
test/glib which has run-test.sh to launch the bus to be tested); or
maybe banging on some of the bindings to get a really smooth approach to
introspection. I could certainly use some help getting to 1.0 sooner and
in a more polished state.

Havoc