Exposing some type details in the introspection XML?

[Matthew Bromley-Barratt]

> >
> > > And I mean:
> >> > a(sa(sa(sa(sgya{sv})a{sv})a(sa(sga{sv})a{sv})a(sgya{sv})a{sv})a{sv}) ?
> >> > that's just so delicious! Just think about how much more awesome
> >> > this signature string could be if there were structures.
> >>
> >> Somewhere, something went horribly wrong...
> >>
> >
> > Feels like quite an intimidating task to define a new IDL for DBus.
> > And I am not convinced the various bindings / code generators will
> > follow, because parsing it will be an additional pain, whereas XML,
> > well, it's there and it's easy to add new attributes/elements for
> > existing code & interfaces. Perhaps the "delicious" giant a{sv} is
> > indeed a better proposal, since it fits nicely. But to me it's just a
> > dbus marshalled version of the XML, and we are back to similar
> > problems/considerations regarding the schema. It may be even less
> > convenient, less flexible, beside being humanly unreadable, and pretty
> shallow.
> >
> > In the past (commit f3549401113b "spec: Allow <annotation> in <arg>
> > elements in introspection XML", or commit dc12fac5f8a36d), we have
> > modified the introspection schema slightly, without bumping version
> > etc. Did it bother anyone? Can we further add elements or attributes
> > here and there, without touching the existing ones?
> >
> > Although I can't find it explicitly in the spec, I suppose the
> > introspection XML is free to introduce extra/new xml namespaces
> > already in a compatible manner, correct?
> >
> > Clearly, it would be inconvenient if both client & servers would have
> > to handle different introspection formats/versions in the future.
> >
> >
> So I toyed with this idea a bit in zbus, and would like to get some feedback
> before I go further. Eventually, I can either try to implement the proposal in
> sd-bus or gdbus. I would like to invite the maintainers of go and python
> bindings to give feedback, and eventually give a try at implementing it, if they
> are interested.
> 
> As we can't extend the current XML without risk of breaking parsers, and we
> would have to open the door to other formats eventually (custom binding
> formats too, perhaps), let's start with a new Introspect() method:
> 
>   <interface name="org.freedesktop.DBus.Introspectable2">
>     <method name="Introspect">
>       <arg name="format" type="s" direction="in"/>
>       <arg name="options" type="a{sv}" direction="in"/>
>       <arg type="ay" direction="out"/>
>     </method>
>   </interface>
> 
> That should be straightforward, a new interface Introspectable2 is added,
> with a new method to introspect, that takes a format as a string (for example
> "dbus.v1"), our idiomatic "a{sv}" dict for options (I think we could have
> common options, such as "dbus.depth", "dbus.only-nodes" etc, and format
> specific options with for example "fmt.comments").
> 
> The returned value is changed to "ay", so it can return anything (although
> most likely utf8 strings).
> 
> I would like to extend the current XML, let's call it "dbus.v2" format for
> example for the above method, or v2 in short. The goal is that a v2 parser
> should be compatible with current v1 XML documents. It should also be fairly
> easy to extend a v1 parser to support v2. Or to generate v2 progressively
> from v1. In other words, I tried to extend the v1 format in some compatible
> manners, so both v1 and v2 can coexist and share code.
> 
> - Comments
> 
> I propose to have <!-- --> comments to be associated with the next element.
> Ex:
> 
>   <!-- This D-Bus interface is implemented by the
>        /org/freedesktop/PolicyKit1/Authority object on the well-known name
>        org.freedesktop.PolicyKit1 on the system message bus. -->
>   <interface name="org.freedesktop.PolicyKit1.Authority"
> 
> We could standardize on the markdown syntax too.
> 
> - New elements
> 
> Since a well-formed XML document should have only one top-level element,
> I propose to allow various new type elements to the top-level <node/>.
> Alternatively, we could have a new <introspect> root element, and have a
> single child <node> there.
> 
> - New type names
> 
> Some proposed elements introduce new type names. I propose to change a
> bit the way the "type" attributes are usually parsed and interpreted. If the
> type starts with a capital, then the alphanumeric characters that follow form
> a type name (that references a type element). Additionally, "[T]" is added for
> array types. No dictionary syntax is necessary. Complex types can be
> represented with <structure> types.
> 
> - <error> element
> 
> This element declares an error domain. It must have a domain name, and a
> type, and list the various errors as childrens. Ex:
> 
>   <!-- Errors that can be returned by various method calls. -->
>   <error domain="org.freedesktop.PolicyKit1.Error" type="s">
>     <item name="Failed"/>
>     <item name="Cancelled"/>
>     <item name="NotSupported"/>
>     <item name="NotAuthorized"/>
>     <item name="CancellationIdNotUnique"/>
>   </error>
> 
> the <interface> gains a new attribute
> error="org.freedesktop.PolicyKit1.Error" to associate the interface methods
> with the error domain.
> 
> - <enum> type element
> 
> Declares an enumeration. It must have a name and a basic type attribute.
> Each item has an optional name and a value. (the name is optional, it can be
> generated by the binding, and we can have enum of strings without
> repeating name/value), ex:
> 
>   <!-- An enumeration for granting implicit authorizations -->
>   <enum name="ImplicitAuthorization" type="u">
>     <item name="NotAuthorized" value="0"/>
>     <item name="AuthenticationRequired" value="1"/>
>     <item name="AdministratorAuthenticationRequired" value="2"/>
>     <item name="AuthenticationRequiredRetained" value="3"/>
>     <item name="AdministratorAuthenticationRequiredRetained" value="4"/>
>     <item name="Authorized" value="5"/>
>   </enum>
> 
>   <!-- The type of the subject. -->
>   <enum name="SubjectKind" type="s">
>     <item value="unix-process"/>
>     <item value="unix-session"/>
>     <item value="system-bus-name"/>
>   </enum>
> 
> - <dictenum> type element
> 
> Declares a dictionary, by listing the well-known keys (and if the set of
> accepted keys is strict), and associated types. If the key type is "v", the items
> should have an associated type too.
> 
> Ex:
> 
>   <!-- Somebody's details. -->
>   <dictenum name="PersonDetails" type="ss">
>     <item value="firstname"/>
>     <item value="lastname"/>
>   </dictenum>
> 
>   <!-- Details about the identity. -->
>   <dictenum name="IdentityDetails" type="sv">
>     <item value="uid" type="u"/>
>     <item value="gid" type="u"/>
>   </dictenum>
> 
> - <struct> type element
> 
> Declares a struct, with field name and type. Ex:
> 
>   <!-- This struct describes a temporary authorization. -->
>   <struct name="TemporaryAuthorization">
>     <item name="id" type="s"/>
>     <item name="action_id" type="s"/>
>     <item name="subject" type="Subject"/>
>     <item name="time_obtained" type="t"/>
>     <item name="time_expires" type="t"/>
>   </struct>
> 
> With the above, it becomes possible to generate much nicer high-level
> bindings. With zbus, we go from v1
> https://gitlab.freedesktop.org/snippets/1117 to
> https://gitlab.freedesktop.org/snippets/1118 (the v2 XML:
> https://gitlab.freedesktop.org/snippets/1119).
> 
> What do you think?
> 
> thanks
> 
> --
> Marc-Andr? Lureau

I certainly see value in dBus supporting the things you've suggested, (structs, etc), but I can also see long term value in implementing them by adopting something else instead of extending the existing IDL. I can't help but feel that, the more the existing IDL schema language and tooling for dBus is evolved, the more it will resemble some other existing standard and its associated tooling, whilst not being able to benefit from them. 

At some point in the future it is possible that it will become widely recognised that a "big bang" is required, that a switch to some existing standard is imperative. If and when that happens, that would be quite a lot of work. However, it may be substantially less work overall than had already gone into patching up the existing IDL and code generator over the years between now and then. So there may be some benefit in thinking what the long term feature set for dBus needs to be, and considering whether or not a big bang now would be the shortest path to delivering those. 

As to what that standard could be, well, there's not many really good ones to pick from. One candidate I feel is Google protocol buffers. dBus could offer the use of Google Protocol Buffers as an alternative to the current IDL / marshalling, providing a standard .proto defining anything that dBus needed to be standardised. A lot of what is being discussed could be easily covered off by expanding that standard .proto file, with a minimal of code changes required to make use of that. There's already a huge number of code generators for every language under the sun (one of gpb's big strengths), and any improvements and additions directly benefit dBus. In other regards gbp is only just suitable - dBus would be depending using the oneof message type to be able to receive typed messages. 

Are there any spare bits in the existing interface to allow a dBus object to support 2 different marshalling formats (e.g. the existing one and, say, GPB)? That would allow for a period of transition.

Some time ago I floated the idea of extending the existing IDL with value and size constraints, generating little enthusiasm! But if there were a move to add to the IDL, rather than switch to something else, it would be excellent to be able to introduce those too. Unfortunately, GPB doesn't allow for size and value constraints, but I'm about to suggest that to that community, see if they like it.

Many thanks,

Matthew Barratt