[systemd-devel] Fwd: CPU time of journal

[Lennart Poettering]

On Tue, 20.12.16 14:43, Shuang Liu (shuangistan at gmail.com) wrote:

> Hello developers,
> 
> Merry Xmas and happy new year.
> I would like to disturb you little bit about the journal behavior.
> 
> We are doing some embedded systems with high boot performance requirement.
> First, thanks to systemd the startup configuration is no more a
> sophisticated job.
> 
> Still, we faced some problems with journald in our work.
> We found the CPU latency for each log is a bit higher than we expected,
> especially when we have thousands of logs during startup, which in fact
> delays our early applications during the startup.
> 
> In a test, we were continuous logging information using different methods
> which journald support, for example syslog.
> We found that more than half of the total CPU cycles (from the flame graph
> we got) are used by function cg_pid_get_path_shifted(). I am not sure why
> this function is little bit too heavy.
> 
> This function seems to be used from two places:
> 
> The first is to test the journal rate limit. In my understanding, if we
> turn off the limit, ie. set it to zero. The function is not needed to be
> called. In fact the check whether it is disabled comes later, which makes
> the function cg_pid_get_path_shifted() still being called. Maybe just by
> preponing the check we could economize CPU cycles.

I figure we could avoid this call entirely, by passing in the
parameter we already got from the first invocaiton.

> 
> The second is to collect the metadata, especially the cgroup information.
> We've tried to bypass the function in this part of source code and are able
> to accelerate the startup.
> 
> One effect is that we lost some metadata in the journal, like _COMM= _EXE=
> _CMDLINE= _SYSTEMD_CGROUP= _SYSTEMD_UNIT= _SYSTEMD_SLICE=
> These information are not critical in our project setting thus we can
> accept the loss of these data.
> 
> I would like to get advice from you, If there are any other risks, from
> which we may get into trouble.

Well, with this you lose out on most of the metadata collection,
i.e. "systemctl status foobar.service" won't be able to filter by unit
anymore. However, if the metadata and filtering is not relevant to you
you can certainly live without it.

Ideally, the kernel would just attach the necessary data directly for
us to the log datagrams. That would fix a race and be much
cheaper. Unfortunately noone bothered so far to make this acceptible
to the kernel maintainers...

> And maybe it is an option if we could have an option in the runtime
> configuration to select which kind of metadata are required to avoid
> unnecessary latency.

I am not sure I like this idea... The primary reason the journal
exists is so that we can implement "systemctl status" properly, after
all...

mybe some hidden env var we check might be OK, but certainly not a
first-level configuration file option...

Lennart

-- 
Lennart Poettering, Red Hat