avahi-daemon exhibits increasing memory consumption due to libdbus-1 FIXME

[Thomas Fitzsimmons]

Hi,

We ship a memory-constrained embedded product that makes moderate use of
avahi-daemon and Avahi utilities like avahi-browse.  We've been tracking
an issue where avahi-daemon's memory footprint grows slowly, linearly
and indefinitely, eventually exhausting all system memory.  I believe
I've discovered the root cause residing in libdbus-1.

Whenever an Avahi utility connects to avahi-daemon through D-Bus,
avahi-daemon registers a new object path for the requested service.  For
example when "avahi-browse -at" is called, avahi-daemon handles the
incoming request like this:

  [receive service request from a new Avahi utility process, ClientX]
  [...]
  client->id = server->current_id++;
  [...]
  i->path = avahi_strdup_printf("/Client%u/ServiceTypeBrowser%u", client->id, i->id);
  dbus_connection_register_object_path(connection, i->path, &vtable, i);
  [...]
  [ClientX disconnects]
  [...]
  dbus_connection_unregister_object_path(connection, i->path);

server->current_id is an unsigned int, initialized to zero on
avahi-daemon startup, never decremented, and only wraps back to zero
after reaching UINT_MAX.  So in effect, a given "ClientX" path element
is never reused.  e.g.:

  register   "/Client1/ServiceTypeBrowser1")
  unregister "/Client1/ServiceTypeBrowser1")
  register   "/Client2/ServiceTypeBrowser1")
  unregister "/Client2/ServiceTypeBrowser1")
  [...]
  register   "/Client325/ServiceTypeBrowser1")
  unregister "/Client325/ServiceTypeBrowser1")

avahi-daemon properly requests that the object path be unregistered.
libdbus-1 properly frees the ServiceTypeBrowser from connection's object
tree.  The issue is that libdbus-1 does not free the "ClientX" struct (a
DBusObjectSubtree) until the connection to dbus-daemon is shut down.
That doesn't happen unless avahi-daemon shuts down or gets disconnected
from D-Bus.

Therefore in the course of normal operation, every ClientX subtree
object persists in memory, referenced by the connection's tree.  The
precise accounting is tricky but every invocation of an Avahi utility
causes an increase in avahi-daemon's memory consumption of at least
sizeof(DBusObjectSubtree) + strlen ("Client...").

Such a small increase is hard to notice on systems that have lots of
memory and use the Avahi utilities only occasionally (e.g., a typical
desktop system).  But our product is memory-constrained and our
applications invoke these tools frequently, so the cumulative increase
becomes significant.  For example on my machine (avahi-0.6.31,
dbus-1.4.16), after avahi-daemon has reached steady-state:

for i in `seq 1 500`; do avahi-browse -at > /dev/null; done

causes an increase of 28 kB to avahi-daemon's resident set size that is
not reclaimed until shutdown.  Higher numbers of iterations produce
higher increases.

Failure to free orphaned DBusObjectSubtree structs is a known limitation
of libdbus-1, documented in this dbus-object-tree.c FIXME dating back to
2003:

  /* If we have no subtrees of our own, remove from
   * our parent (FIXME could also be more aggressive
   * and remove our parent if it becomes empty)
   */

We've tested a workaround that patches avahi-daemon to instead register
paths of the form "/Client%uServiceTypeBrowser%u", so that no parent
struct is allocated.  This eliminates the constant increase in memory
consumption.

This libdbus-1 issue has been present for a long time and seems low
impact.  And it isn't technically a memory leak because the parent
subtree objects are always reachable and are properly freed when the
connection shuts down.  But under the right conditions -- a long-running
process, a long-lived connection to D-Bus, registration and
unregistration of an unbounded number of unique object paths -- it can
create a significant problem in real-world situations (e.g., in our
product).  It may impact other services that use libdbus-1.

Would the D-Bus maintainers be open to a patch to fix this long-standing
FIXME?

Thomas