D-Bus daemon hits 100% CPU

[Simon McVittie]

On 03/06/16 17:13, marcin at saepia.net wrote:
> I've observed weird behaviour with reference D-Bus daemon (version from
> ubuntu 16.04), it ocassionally hits 100% CPU.

What exact version is this?

On Debian derivatives like Ubuntu, if you are using the packaged
dbus-daemon installed at /usr/bin/dbus-daemon, run "dpkg-query -W dbus".

> It seem to happen when it
> gets around 1000 client connections

1000 client connections sounds suspiciously similar to 1024, which is a
common value for the RLIMIT_NOFILE limit that the kernel imposes on the
dbus-daemon (RLIMIT_NOFILE, "Max open files" in /proc/${pid}/limits). We
had issues a few years ago with the dbus-daemon's event loop going into
a busy-loop when it tried to poll() RLIMIT_NOFILE file descriptors, but
there are two reasons why we believe that should no longer apply:

* that bug was caused by adding each file descriptor to the event loop
  twice separately (for reading and writing), and we changed it to merge
  the two watches into a single poll record (fixed in 1.5.6, bug
  #23194);
* we shouldn't be using poll() on modern Linux anyway, because I added
  epoll support around the same time (1.5.10, bug #33337)

I'd be interested in seeing stack traces and/or strace output from the
busy-loop. Please open a bug. If you are using the packaged dbus-daemon
from Ubuntu, install dbus-1-dbg before getting your stack traces, so
that they'll contain useful information.

> over TCP

The TCP transport has the same security properties as 1980s-era NFS: it
is only as secure as your network's ability to resist eavesdropping and
spoofing, i.e. in practice it cannot be assumed to be secure. Please use
the default AF_UNIX sockets instead. If you need remote D-Bus, I would
recommend forwarding the normal AF_UNIX socket using a ssh tunnel or
socat or similar.

(One exception: on Windows, there is nothing better than TCP available
to us, so it is OK to use the TCP transport listening on 127.0.0.1 or
::1; however, we do not consider dbus on Windows to be suitable for use
at a security boundary.)

This has been a serious real world problem: a dbus-daemon inadvisably
configured to use the TCP transport was a key part of the 2015 Jeep hack
that led to a recall of 1.4 million vehicles. We have seriously
considered making dbus-daemon refuse to listen on TCP at all, except for
127.0.0.1 and ::1, to make it harder for dbus users to shoot themselves
in the foot, and I might still do that for 1.12.

> Is there anything particular in the config that could limit its capacity
> regarding max clients limit?

You haven't told me what configuration you are using, or whether this is
the system bus, the session bus or a custom bus, so I can't say. If you
are using TCP, then you are using a non-default configuration of some
sort. What is that configuration?

There are some hard-coded default limits used if the bus configuration
(session.conf, system.conf, --config argument) does not specify a limit,
but none of those are close to 1000: up to 2048 connections, of which
each uid can have up to 256.

-- 
Simon McVittie
Collabora Ltd. <http://www.collabora.com/>