[systemd-devel] sd-bus: serving method call message in a separate thread

[Stanislav Angelovič]

Hi sd-bus-ers!

Quick question: How can I process a method call on service side in a
different thread, without creating race condition?

Longer version:

In sdbus-c++, we are working on server-side asynchronous method call
support.

In sd-bus, a service handles D-Bus method calls via
sd_bus_message_handler_t callback that is registered during object vtable
registration. The callback receives sd_bus_message* as the first parameter
(among others), which is the method call message. In a typical
implementation, this method call is served synchronously (D-Bus method
parameters are deserialized from the message, an operation is executed, and
either method reply or method error, depending on the condition, is sent
back) and the callback returns execution to sd-bus. Standard stuff so far.

Now, I would like to handle the method call asynchronously, in a thread
pool. So a typical approach would be to increment ref count of
sd_bus_message and push the sd_bus_message* in a queue. It would then be
popped by one of worker threads and processed -- method parameters are
deserialized, method logic is executed, and either method reply or method
error is created and sent back, and sd_bus_message* ref count is
decremented. Now, sending back the reply or an error is thread-safe in my
implementation, I don't have an issue with that. The problem that I want to
discuss is the method call sd_bus_message instance. I cannot simply forward
a pointer to it to a separate thread, because of it's non-atomic ref count
handling, which is the source of race condition (hopefully this is the only
data race condition here). It's manipulated by both the D-Bus loop thread
(which decrements ref count after sd_bus_message_handler_t returns) and the
worker thread concurrently (which also decrements ref count after it has
handled the method call).

I see three potential solutions now:

1. In sd_bus_message_handler, create a deep copy of the method call
sd_bus_message via sd_bus_message_copy(). And pass a pointer to this copy
to a worker thread. This is very straight-forward and simple, it solves the
race condition, but introduces a performance cost of copying (I don't know
how big this cost is, perhaps it's rather negligible).

2.  Don't pass the sd_bus_message* to the worker thread. In
sd_bus_message_handler, rather deserialize all arguments, create (empty)
method reply message, and move these to the worker thread. The worker
thread executes the logic and serializes results to that reply message, and
sends it back. The problem here is that we have to create a method reply or
method error before the fact (before executing method logic), which in case
of method error is impossible because we don't know possible error name and
message beforehand.

3. Solution on sd-bus side :-) which is to make sd_bus_message ref count
handling thread-safe (atomic), just like one for sd_bus. This avoids the
need for deep copy. What do you think? Anyway, we'd still have to come up
with a solution now, based on already releases sd-bus versions.

So the only feasible solution for now seems to be #1 (while #3 could also
be a sd-bus improvement for the future). Do you agree that this is the good
way along the lines of sd-bus design principles, or are there more options
I'm unaware of?

Thank you for your constructive comments, and sorry for long elaboration :)
I wanted to be clear...

Cheers,
Stanislav.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20190304/81e266d0/attachment-0001.html>