D/BUS IDL compiler ...

[Havoc Pennington]

Hi,

On Fri, 2004-03-05 at 11:23, Michael Meeks wrote:
> 
> 	So - attached is my 1st pass at a dbus IDL compiler

My thought was to only do IDL for Qt/GLib bindings, and to use the
native object representations, IDL formats, introspection data, and so
forth for those such as they exist.

For GLib I wanted to do it the same way Qt works, that is:
 - parse a regular GObject to obtain introspection and marshaling data
   sufficient to invoke its methods
 - store this introspection/marshaling data in a variable
 - dynamically dispatch incoming DBusMessage based on said data

The key insight discussed on this list before is that moc is not in fact
a preprocessor; it's a metadata extractor. It creates a variable in the
final C++ file containing dynamic introspection information.
That's why in Qt you have dynamic introspection without writing an IDL
file.

Obviously to do this with GObject we would require following some fairly
strict coding conventions, e.g. you could not name your objects
incorrectly or use types that don't map to dbus types.

Part of this code already exists in the dbus/glib subdir, there's no C
scanner/heuristic-parser though.

The way I was going to do it is:
 - there's an XML introspection format returned by Introspect()
   (effectively an IDL, but can include other metadata too)
 - have the C scanner/heuristic-parser scan the C file and generate
   said XML
 - have a second pass that converts the XML representation to a 
   metadata variable that can be included in your C
 - have a method you call in class_init to register the metadata
   variable (see dbus-glib.h dbus_g_object_class_install_info)
 - have a method you call to register any GObject with a DBusConnection,
   which takes advantage of this info

I'm reciting this from memory, I may have posted about it to the list
earlier and I probably got it closer to correct at that point.

I'm not sure anyone but me has ever understood what I mean here, but
maybe the missing link is to look at exactly what moc output contains;
Matthias Ettrich gave a nice presentation at Nove Hrady on that that I
saw.

This is all on the *server* side - for the client side, I was just using
varargs, as the typesafe C function wrappers create makefile complexity
and code size. See dbus_gproxy_*

If you did typesafe wrappers I'd do those as an opaque typedef that was
cast back to DBusGProxy inside the wrapper.

This is all for GLib of course. The idea is to avoid a DLib and DType
and so forth. 

> ; so far it parses a
> chunk of (what looks like) pseudo-IDL pinched from the HAL spec

If the IDL format is CORBA/COM-like but different, maybe we should try
to just use one of those existing styles.

If making something up, one option is to use the Introspect() XML
format.

> 	* It's written in perl - I don't like perl - perl, being a 
> 	  pre-requisite for autotools is available on every developer
> 	  platform; the IDL only needs re-compiling by developers. QED.
> 	  Indeed, perl is (empirically) more portable than autotools,
> 	  it's also an excellent language for string processing, and
> 	  made it very fast to write this. It's also not that bad if
> 	  you think for a few seconds before typing.

Perl seems fine to me for this, though I would not personally be much
good at editing it. ;-)

> 	* It appears to re-implement dbus_message_append_valist etc.
> 	  however - it needs to do this to return the advanced va_list
> 	  to allow return value de-marshalling.

Maybe there is a reasonable way to extend the API to avoid this?

> 	* It doesn't implement the array types - correct, the lack of a
> 	  decent recursive type description is sufficiently annoying to
> 	  me that I can't stomach it yet.

One idea is to use some _other_ type description, e.g. the one from
GLib, Qt, CORBA... in fact simply replicating a subset of the CORBA API
and IDL and tunneling it through here may be one approach to what you
want.

Remember that the design goal is to stop D-BUS type system right off the
wire and immediately convert to a native type system. 
i.e. basically what you do in .NET remoting or when using XML-RPC or
something. Avoiding DVariant, DType, D* is a deliberate choice, though
not a finished or well-proven one at the moment.

> 	* The license is LGPL/X11 - I'm still deeply concerned with the
> 	  unresolved AFL legal issue.

It's in the works, the opensource.org guys just haven't put out the new
AFL version. If D-BUS were at all ready from a _technical_ standpoint
I'd feel more urgency ;-) see TODO

> 	* Why do we have a 'service' and a 'path' concept ? surely a 
> 	  given service will bind to a given path ? someone must know
> 	  this ? it's slightly surreal having a method taking three
> 	  strings which are ~mostly identical in ~every case. Is it
> 	  really worth having this extra dimension to the space ? is
> 	  it really orthogonal ?

There's been previous discussion of this, for example here:
http://freedesktop.org/pipermail/dbus/2003-September/000483.html

Note that in DBusGProxy the strings are just specified when you create
the proxy, i.e. a proxy is for an instance not a type.

One issue is how we map to DCOP which has all three concepts. A
technical issue that needs resolving before we fully finalize DBUS is to
actually develop proof of concept DCOP mapping.

> 	* 'signals' - I'm confused as to what to generate for these;
> 	  it seems to me that there are 2 needs:
> 
> 		a) marshalling type information for the emitter
> 		b) proxying type information for the listener
> 
> 	  of course - we could duplicate this information in some
> 	  way - but given that a 'signal' seems to be (simply) a 
> 	  oneway method call; it would seem more sensible (perhaps)
> 	  to use a 'Listener' pattern to allow a simpler
> 	  implementation.

We're mapping to GObject/QObject which both have signals, is one reason
for this. Well, as does C# (sort of). So here we can more
obviously/directly map the nature of the native object.

There was some waffling on this point, signals were added later; it's
long enough ago that I don't remember all the issues.

dbus-glib.h/dbus-gproxy.c shows my most recent line of thinking on what
code to generate on client side for signals. On the server side, was
simply going to forward on the GObject signals, but probably only those
with a magic comment /*< remote >*/ or something along those lines. i.e.
certain marked signals would be included in the metadata scanned by the
moc-equivalent.

> 	* signal emission - in Bonobo we discovered that often
> 	  signals are emitted with no listener :-) thus it makes
> 	  sense to be able to detect this eventuality from the	
> 	  emitter and avoid the cost - is that something that makes
> 	  sense with dbus ? and/or that is done already.

I spent a lot of time on the "how to create and store a signal
connection" question, there are lots of tradeoffs. Right now the emitter
always sends the signal to the bus (or peer in busless connections), but
the bus only forwards to apps that have registered to receive the
signal.

There may be some old threads about this in the archives, I'm not sure.

I think I eventually decided we could optimize this later if needed.

> 	My hope is - that this will be something useful; I guess it'll be more
> useful on the server-side though - at least, judging from my quick
> skimming of HAL there is too much hand-coded error-prone
> marshal/de-marshal logic around that's verbose & hard to audit. 
> 
> 	I hope to do a little work on this in the next few days, so flames,
> outrage, constructive comments etc. appreciated :-) and/or is this
> something that could live in dbus CVS ?

It's OK with me to put in CVS in a subdirectory, with the same
--enable-foo option to configure almost all the current subdirectories
have; i.e. basically treat this as another binding (well, it _is_
another binding). Main challenge I guess would be to think of what to
call it ;-)

Havoc