[systemd-devel] [RFC] [PATCH 0/3] journal: Add deferred log processing to reduce synchonous IO overhead

[Karol Lewandowski]

On Thu, Dec 12, 2013 at 03:51:05PM +0100, Lennart Poettering wrote:
> On Thu, 12.12.13 14:12, Karol Lewandowski (k.lewandowsk at samsung.com) wrote:

(OP here from private email.)

> > Folks,
> > 
> > We are trying to apply journald as default (and only) logging system
> > for our ARM-based handhelds.  There is one major problem, though -
> > performance.
> > 
> > We have found that journald can consume considerable amount of CPU 
> > time under moderate load (10-20% of CPU per, for say 100-300msg/s).
> > 
> > We have profiled journald and there are few places that could benefit
> > from _some_ optimizations but I have come to conclusion it isn't
> > journald code that is real bottleneck - it's, or seems to be,
> > synchronous nature of send()/recv().  In default configuration each
> > send() can cause context-switch which is especially costly on ARM.
> 
> Hmm, but this wouldn't be any different from old syslog, right? I mean,
> old syslog also uses sendv() and recv() on AF_UNIX SOCK_DGRAM sockets...

That's true, however, we aren't migrating from syslog, but from
Android Logger.  Android provides /dev/log_{main,radio,...} device
nodes which are simple in-kernel circular buffers apps can write to at
any time.  This was invented when /dev/kmsg wasn't writable and serves
more-or-less same purpose.

>From 2.6.29 it's in mailine kernel under drivers/staging/android/logger.c


> Note that on Linux the number of datagrams you can queue in an
> AF_UNIX/SOCK_DGRAM socket is very low (15). THis is controlled via
> /proc/sys/net/unix/max_dgram_qlen. When this limit is hit the client
> needs to wait for the server to finish processing. Unfortunately there's
> only that system-global way to increase the qlen. For a long time it has
> been on our wishlist to make this a per-socket tunable with a new
> sockopt() that behaves similar to SO_SNDBUF and friends...

We have played this tunable and it didn't bring considerable/measurable
improvements, unfortunately.

 
> > This is why I decided to try to make logging completely async and see
> > if/what changes.
> 
> Well, if AF_UNIX is slow, then we really should try to fix that in the
> kernel instead of bypassing it in userspace... Whether we rely on the
> socket buffer of the AF_UNIX socket or put together our own buffer in
> /dev/shm shouldn't be much of a difference if they both have the same
> size and can take the same number of entries...

I do agree.  Truth is I have looked at linux/net/ code once but I
didn't grok it well.  I guess it's the time to take a second look. ;)

Cheers,
Karol