runaway memory with dbus-daemon --session

[Simon McVittie]

On Wed, 28 Jul 2021 at 13:02:23 -0400, David H Durgee wrote:
> My system was running slow Monday afternoon. Checking the system monitor I
> found that the dbus-daemon process noted above was using 502.9MB of memory! Not
> knowing how else to correct this, I closed out all my applications and
> restarted my system. After the restart this process is using 1.6MB of memory.

I suspect something flooded it with messages that couldn't be delivered
immediately. In this situation the messages get queued up in the
dbus-daemon; if the destination is not receiving but remains connected,
they'll stay in memory indefinitely. The system bus tries to defend
itself against this with arbitrary limits (quota), but the session bus
is designed to be used between trusted processes, so its limits are
essentially infinite.

Enforcing the limits is not as easy as it sounds, because if they're
set low, then processes can carry out denial-of-service attacks on each
other by tricking another process into using up its quota.

The semantics of D-Bus do not make this easy either, because the only
two things we can do without allocating memory are dropping a message on
the floor or disconnecting a client, neither of which give a client an
opportunity to detect it and recover.

https://gitlab.freedesktop.org/dbus/dbus/-/issues/36 (probably easier to
read at the original https://bugs.freedesktop.org/show_bug.cgi?id=33606)
is closely related to this.

In particular, https://bugs.freedesktop.org/show_bug.cgi?id=33606#c11
outlines some of the ways a denial-of-service situation can occur.

dbus-broker, a Linux-specific reimplementation of the reference
dbus-daemon, tries to resolve that with two main techniques: it makes
use of Linux-specific socket features, and it tries to assign "blame"
for each message so that the right process gets punished for attempts
to DoS the system. I haven't reviewed its idea of which processes get
"blamed" for which messages, and they might not be right for all
client/service pairs.

A related issue is that in order to increase message latency and
throughput under normal circumstances, dbus-daemon uses a memory pool for
some data types which does not return all memory to the OS, so when it
has reached a high level of memory use it will tend to stay there. The
reason why this has not been removed is that nobody (that we know of)
has a realistic benchmark for dbus latency and throughput under normal use,
so we do not have a good way to assess the performance impact of removing
the memory-pool implementation.

> Is there a way to cleanly restart this process short of a full system shutdown/
> restart?

Not really. You can log out from all your login sessions and log back in,
but that's approximately the same level of disruption as rebooting for
typical single-user systems.

    smcv