libdbus _dbus_connection_block_pending_call blocks until timeout, even after reply is received

[Simon McVittie]

On Fri, 15 Dec 2017 at 10:38:51 +0000, Manish Narang wrote:
> We are facing a problem that we believe to be caused by libdbus in a
> multithreaded environment.

libdbus is *meant* to be thread-safe, but is known to have
threading-related issues, so if you use it in a multi-threaded way you
will have to be prepared to debug issues like this.

If you are using libdbus, the low-level reference implementation of D-Bus,
then I would recommend using it in a single-threaded way, with an event
loop doing all D-Bus asynchronously, and other threads posting requests to
the main event loop if necessary. This matches how typical single-threaded
X11 and Wayland graphical toolkits like GTK+ and QtGui must be used. The
same thread that is doing D-Bus can usually also do the GUI and networking
(if present), as long as everything that can block is done asynchronously.

Using libdbus in an asynchronous way requires setting up callbacks for
timeouts, watches, changes to the dispatch status, and waking up the
main loop, for each DBusConnection and each DBusServer - AT-SPI and
QtDBus are among the projects that have suitable code to do this for
the GLib and Qt main loops respectively, which can be used as examples.
I would strongly recommend using an existing main-loop abstraction like
the ones in GLib, Qt, libevent, libsystemd and comparable libraries
rather than making your own.

GDBus (part of GLib) has a design that is better-suited for heavily
multi-threaded use: it has a dedicated thread that does all the actual
D-Bus message sending and receiving, and all other threads post requests
to the D-Bus thread, and get callbacks scheduled in the appropriate
thread when there is a reply. That might be another option.

> The watchdog implementation in our system detected a deadlock in an application
> thread and killed the process. Upon examining the backtrace it was found that
> the process was stuck on a dbus_connection_send_with_reply_and_block call. 
> What is perplexing is that the call is waiting for a reply that has already
> been processed by another thread in the process. 

This might be similar to, or the same root cause as,
<https://bugs.freedesktop.org/show_bug.cgi?id=102839>
which also has more general information on libdbus and threading.

    smcv