[systemd-devel] High CPU usage of journald

[Holger Hans Peter Freyther]

On Mon, Feb 18, 2013 at 08:47:44AM -0800, David Strauss wrote:

Dear David, all,


> It's possible that a lot of it could also allocate on the stack or use
> stack-style management of a pool in heap. This is pretty
> performance-critical code, and we're seeing similar CPU overhead.

I think upstream doesn't care about the CPU overhead of the journald. I
had a quick look on my notebook with the same perf command and _int_malloc
is noticable in the trace as well.

In OpenBSC/Osmocom we have our struct msgb (a clone of struct skb) and
we can pull and push from the allocated memory. E.g. to replace a LayerX
header or prepend another header.


I browsed through journald-server.c:dispatch_message_real and there are
several things one could do... there are certainly a lot more items.


1.) Several things are done over and over again... e.g. calls to the
have_effective_cap.. it is unlikely that it will change after the journald
has started.


2.) Avoid allocation part one...

                r = get_process_cmdline(ucred->pid, 0, false, &t);
                if (r >= 0) {
                        cmdline = strappend("_CMDLINE=", t);
                        free(t);

                        if (cmdline)
                                IOVEC_SET_STRING(iovec[n++], cmdline);
                }

So instead of the strappend (which will do strdup..) one could make sure
that 't' has enough space to prepend the _CMDLINE= or the _EXE, or the
other strings.


3.) Looking at code like... get_process_cmdline

  asprintf(&p, "/proc/%lu/cmdline", (unsigned long) pid)


So first of all /proc/%lu/ is created over and over again for exe, comm,
cmdline, sessionid, cgroup... With a different API one could easily avoid
this.

4.) Still at the same method..

        if (max_length == 0) {
                size_t len = 1;
                while ((c = getc(f)) != EOF) {
                        k = realloc(r, len+1);
                        if (k == NULL) {

is most likely not the most effective way to handle the allocation. Or
maybe call get_process_cmdline with a max length?


5.) I wonder if some of the information could be sent from the systemd
to avoid the work in the journald...



I will end up commenting out most of dispatch_message_real and check if
the CPU down drops to a low single digit. I would assume it will.


regards
	holger