[systemd-devel] sd-bus: serving method call message in a separate thread

[Giacinto Cifelli]

Hi Stanislav,

On Mon, Mar 4, 2019 at 9:56 PM Stanislav Angelovič
<angelovic.s at gmail.com> wrote:
>
> Hi sd-bus-ers!
>
> Quick question: How can I process a method call on service side in a different thread, without creating race condition?
>
> Longer version:
>
> In sdbus-c++, we are working on server-side asynchronous method call support.
>
> In sd-bus, a service handles D-Bus method calls via sd_bus_message_handler_t callback that is registered during object vtable registration. The callback receives sd_bus_message* as the first parameter (among others), which is the method call message. In a typical implementation, this method call is served synchronously (D-Bus method parameters are deserialized from the message, an operation is executed, and either method reply or method error, depending on the condition, is sent back) and the callback returns execution to sd-bus. Standard stuff so far.
>
> Now, I would like to handle the method call asynchronously, in a thread pool. So a typical approach would be to increment ref count of sd_bus_message and push the sd_bus_message* in a queue. It would then be popped by one of worker threads and processed -- method parameters are deserialized, method logic is executed, and either method reply or method error is created and sent back, and sd_bus_message* ref count is decremented. Now, sending back the reply or an error is thread-safe in my implementation, I don't have an issue with that. The problem that I want to discuss is the method call sd_bus_message instance. I cannot simply forward a pointer to it to a separate thread, because of it's non-atomic ref count handling, which is the source of race condition (hopefully this is the only data race condition here). It's manipulated by both the D-Bus loop thread (which decrements ref count after sd_bus_message_handler_t returns) and the worker thread concurrently (which also decrements ref count after it has handled the method call).
>
> I see three potential solutions now:
>
> 1. In sd_bus_message_handler, create a deep copy of the method call sd_bus_message via sd_bus_message_copy(). And pass a pointer to this copy to a worker thread. This is very straight-forward and simple, it solves the race condition, but introduces a performance cost of copying (I don't know how big this cost is, perhaps it's rather negligible).
>
> 2.  Don't pass the sd_bus_message* to the worker thread. In sd_bus_message_handler, rather deserialize all arguments, create (empty) method reply message, and move these to the worker thread. The worker thread executes the logic and serializes results to that reply message, and sends it back. The problem here is that we have to create a method reply or method error before the fact (before executing method logic), which in case of method error is impossible because we don't know possible error name and message beforehand.
>
> 3. Solution on sd-bus side :-) which is to make sd_bus_message ref count handling thread-safe (atomic), just like one for sd_bus. This avoids the need for deep copy. What do you think? Anyway, we'd still have to come up with a solution now, based on already releases sd-bus versions.
>
> So the only feasible solution for now seems to be #1 (while #3 could also be a sd-bus improvement for the future). Do you agree that this is the good way along the lines of sd-bus design principles, or are there more options I'm unaware of?
>

In my use of sdbus I am going for option 3. Also because I am not sure
what happens when sdbus writes on the bus in parallel two long
messages. I know that sdbus is thread aware, but not thread safe.
So 2 long messages could in theory be sent in collision. Maybe you can
verify this opening two (or more) instances of a client that query
automatically all introspectable methods of a specific server
implemented with your thread pool, and see what happens in repeated
trials.

> Thank you for your constructive comments, and sorry for long elaboration :) I wanted to be clear...
>
> Cheers,
> Stanislav.
>

Cheers,
Giaciinto


> _______________________________________________
> systemd-devel mailing list
> systemd-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel