Compatibility between D-Bus and kdbus

[Philip Van Hoof]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 23/10/2014 14:09, Simon McVittie wrote:
> On 22/10/14 22:59, Philip Van Hoof wrote:
>> This was once put in place to guard ourselves against it:
>> 
>> https://git.gnome.org/browse/tracker/tree/src/tracker-store/tracker-resources.vala#n50
>
>> 
> Looks

> as though you're currently limiting yourselves to 10 base-10 
> megabytes (i.e. MB not MiB) of GVariant, whatever that turns into
> in D-Bus serialization[1], plus the message header (which in
> practice shouldn't be more than a couple of KiB even if your object
> paths are really long).
> 
> Is this limitation a problem for you "in real life" (i.e. do you
> ever accidentally exceed it)?

We have no control of what the user will put in the database, so the
practical 'in real life' problem is that if the user does a query that
yields a large result, that it happens.

Of course is this among the reasons why we didn't ever make the
Resources1's Query D-Bus method an officially endorsed public one.

The Query D-Bus method on Steroids1, which is FD passing based, is the
public one which we expose over D-Bus. Once we have the FD passed we
just vmsplice it over pipe of course. That has no such limitations.

The actual only public API we endorse is the libtracker-sparql one.
That one has a direct-access and fallback mechanism that uses the FD
passing D-Bus (never the Resources1-Query message based one).

> Would it be a problem if the limit was (slightly under) 2 MiB
> instead, as it is for broadcasts on kdbus?

For our broadcast and signal needs we have no problems. This isn't
about the GraphUpdated signal. Apologizes if my E-mail was not clear
on that.

> Are you able to split messages arbitrarily, or do you need to
> coalesce some minimum number of events to happen atomically?
> 
> Your message seems to be of signature a(iiii)a(iiii), so, two
> arrays

So that signature is for the GraphUpdated signal where we don't have
any problem. I don't think we'd ever need 2MiB for signals or
broadcast stuff as we pass arrays of 32bit integers representing the
data instead of the data itself (we have functions at the level of
SPARQL to convert the IDs to data and the other way around).

> of 16-byte structs. I think that means you can have up to 131072 
> insertions or deletions packed into a 2 MiB message, minus a few
> to allow for the array lengths and the header; 100K insertions or 
> deletions would certainly be OK, and seems like "enough for anyone"
> :-)

Yep. We have a limit of about 5000 of them in GraphUpdated which we
rarely if ever encounter. The problem is in Query on Resources1.

For Query we have no control over the size of the message to return.

And as mentioned earlier we have a FD passing solution for that instead.

I don't consider Resources1's Query D-Bus method to be official public
API and chances are relatively big that we will remove it some day
and/or only enable it when tracker-store gets started with --debug.

We just have to hunt github with searches to check if really nobody is
using it in production. Before my previous hunt the nemomobile group
still was, for example, for their Gallery application on the Jolla.

> [1] D-Bus and GVariant have slightly different overheads, but
> should be the same order of magnitude

Yep

Kind regards,

Philip



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (MingW32)

iQEcBAEBAgAGBQJUSQHyAAoJEEP2NSGEz4aDHskH/jdh3DO7gja2PNXqYsDmuZ9P
4LN5eHeE48/ZyuGWPrzh4vw/kC0vtlLGt7DNb9XUGfY0HKNKdbmIJQZtotPEKiup
1RWqJqAmWyo++4SLTkGd6iqgXPa5M1bA3eUQ9Zh61foBagdoz0NQkQfr2K0SIy6/
NRE7kk8qse/Xj5SRqeZuNBtyux04N3PlJlRSMhPXs9AiTxLI+jb85v+pkco1I8V6
6Yv7/Q8jrwgfHUig2OV+S3IzR9Jfdnlxb/NiJBmCbj2gyFh4UnVm3BmrAitlgB+O
vI/+/f9yyot3jixo8cQjlYGP22EX3Mngm4P+rx1HwJQlMkqT7uTpqlXvfYc0HDg=
=KpMU
-----END PGP SIGNATURE-----