[systemd-devel] [RFC] [PATCH 0/3] journal: Add deferred log processing to reduce synchonous IO overhead

[Lennart Poettering]

On Thu, 12.12.13 14:12, Karol Lewandowski (k.lewandowsk at samsung.com) wrote:

> Folks,
> 
> We are trying to apply journald as default (and only) logging system
> for our ARM-based handhelds.  There is one major problem, though -
> performance.
> 
> We have found that journald can consume considerable amount of CPU 
> time under moderate load (10-20% of CPU per, for say 100-300msg/s).
> 
> We have profiled journald and there are few places that could benefit
> from _some_ optimizations but I have come to conclusion it isn't
> journald code that is real bottleneck - it's, or seems to be,
> synchronous nature of send()/recv().  In default configuration each
> send() can cause context-switch which is especially costly on ARM.

Hmm, but this wouldn't be any different from old syslog, right? I mean,
old syslog also uses sendv() and recv() on AF_UNIX SOCK_DGRAM sockets...

Note that on Linux the number of datagrams you can queue in an
AF_UNIX/SOCK_DGRAM socket is very low (15). THis is controlled via
/proc/sys/net/unix/max_dgram_qlen. When this limit is hit the client
needs to wait for the server to finish processing. Unfortunately there's
only that system-global way to increase the qlen. For a long time it has
been on our wishlist to make this a per-socket tunable with a new
sockopt() that behaves similar to SO_SNDBUF and friends...

> This is why I decided to try to make logging completely async and see
> if/what changes.

Well, if AF_UNIX is slow, then we really should try to fix that in the
kernel instead of bypassing it in userspace... Whether we rely on the
socket buffer of the AF_UNIX socket or put together our own buffer in
/dev/shm shouldn't be much of a difference if they both have the same
size and can take the same number of entries...

Lennart

-- 
Lennart Poettering, Red Hat