Questions ad marshalling multi-dimensional arrays

[Simon McVittie]

On Thu, 23 Jun 2011 at 12:13:48 +0200, rony wrote:
> while contemplating the marshalling of multidimensional arrays

Just like C (and for that matter, computer memory), D-Bus does not really have
multidimensional arrays, but you can emulate them.

If your underlying data model has multidimensional arrays, you can either:

* use an array of arrays (aaX for some type X):

  1 2 3
  a b c    ->     [[1,2,3], [a,b,c], [x,y,z]]
  x y z

  or if you prefer, [[1,a,x], [2,b,y], [3,c,z]]

* use a single large blob of data in an aX (for some type X), and transmit
  the "stride" out-of-band (e.g. as a separate argument, or implied by your
  API)

  1 2 3
  a b c    ->    array = [1, 2, 3, a, b, c, x, y, z], stride = 3
  x y z

  or if you prefer, array = [1, a, x, 2, b, y, 3, c, z], stride = 3

See any document describing packing multi-dimensional arrays into memory
for more details.

>     * Questions:
>           o Unassigned elements (like arr[1,0], which in some languages
>             could be marked explicitly as NULL/.nil) cannot be dropped
>             (left-out) ?

If you use arrays-of-arrays, then omitting trailing absent elements is
certainly technically possible. You could choose to allow or forbid it
in your API. If allowed, it might look like:

  1 2 3
  a        ->   [[1,2,3], [a], [x,y]]
  x y

If you use a single large data blob, you can do the same (but in a more
limited way - you could only omit trailing columns from the last row, or
vice versa if you use the opposite layout) by having its length not be
divisible by the stride:

  1 2 3
  a b c    ->    array = [1, 2, 3, a, b, c, x], stride = 3
  x

... but if you want to forbid this for simplicity, nothing is stopping you.

Basic types do not have any built-in null value, so you will have to reserve
one. "" and 0 are reasonable choices, if you don't need a distinction between
"" and NULL, or between 0 and NULL.

(For instance, in C, strings and other pointer types can have a distinction
between "" and NULL in contexts where NULL is allowed, but integers cannot.)

Note that "" is not a valid object path. "/" is, and Telepathy uses this as a
pseudo-NULL object path in a couple of places.

When marshalled, D-Bus arrays come out as the array length, followed by
the items in sequence. So, using arrays of arrays results in larger
messages than using a single array and communicating the stride separately,
if that matters to you.

>           o Which would be the marshalling sequence

You can choose whether to use "row-major" or "column-major" ordering.
Whichever you do, document it as part of your API specification, preferably
with an example.

>           o Given the above example, would it be o.k. to leave out
>             "trailing empty elements of the array"?

You can define it to be OK. If you do, document it as part of your API
specification, preferably with an example.

>           o Is it possible to marshall an empty array by merely using a
>             sequence of open_container(DBUS_TYPE_ARRAY) - no-content -
>             close_container?

Yes. Open the container and immediately close it.

> P.S.: Are there perhaps any thoughts about defining an explicit
> NULL/.nil value (maybe at least transportable in the context of variants) ?

This is more difficult than it looks because it's an incompatible change to
the type system.

In the proposed "maybe" type, mX means "maybe X" for any type X; it can
either have value Just V (where V is any valid value for X),
or Nothing (analogous to NULL). It's implemented as an array whose length is
at most 1. See <https://bugs.freedesktop.org/show_bug.cgi?id=27857>. GVariant
(a superset of the D-Bus type system) implements this, but D-Bus doesn't.

One alternative scheme is to allow DBUS_TYPE_INVALID in variants. GVariant
doesn't implement this, it'd still be an incompatible change (so it would be
impossible to send Variant(nil) to older implementations), and as far as
I know, nobody has implemented this or given a solid reason why it's
necessary, given the alternatives.

Another alternative scheme is to allow the trivial struct type, "()" (again,
GVariant allows this, but D-Bus does not currently), and treat the single
value of that type as if it was NULL when it's seen in variants.

Another alternative scheme is to declare that by convention, a certain
set of variant contents (perhaps Variant(Variant(False)) or something)
is treated as if it was NULL. You could put this in an API already, if you
need it.

    S