[systemd-devel] How to scale journald correctly?

[Lennart Poettering]

On Di, 10.07.18 14:24, Martin Häcker (mh at sntl-publishing.com) wrote:

> 
> To work around this, we continued by increasing SystemMaxFileSize=20G to
> limit the amount of files journald would produce. This however led to
> the unexpected behaviour of journald exhausting the IO bandwith of the
> server it is running on. Documented here:
> <https://bugzilla.redhat.com/show_bug.cgi?id=1599658> Interestingly this
> setting seemed to work well for some hours, but then suddenly let
> JournalD use up all IO resources of the system.
> 
> Which begs the question: What are the limits to file sizes, total log
> size, log intake rate, ... that journald is engineered to work well
> with? What can be reasonable values for these Settings? Where there any
> changes in recent releases that could affect this (not that I particular
> want to upgrade a core component of my CentOS systems).

Yes, the journal has received a ton of improvements in all areas since 219.

In general: splitting up too heavily makes journalctl grow with O(n)
where n is the number of split out files. This is the main performance
problem right now. THe O(n) thing is fixable, but so far nobody spent
the time to fix it fully.

journald is designed to just write out what is being thrown at it,
with the speed the underlying device permits. It is using relatively
large reception buffers for this, so that clients don't have to stall
on journald. If you flood journald heavily it will eventually cause
clients to block though.

In more recent journal versions (i.e. much newer than 219) log
metadata is cached, which will increase performance substantially
under load.

Also note that clients can slow journald considerably, if they for
example log at needlessly high log levels (that's because journald
will sync() after each append when it sees messages of CRIT, ALERT,
EMERG level...

journald's write logic is heavier than let's say syslog, as it builds
a bisection table for each field, as it adds entries to the end of
journal files, however, this shouldn't be overlay heavy.

I am not aware of any work on performance measurements for the
journald, and figuring out in detail how things scale, and were
improvements need to be made with greatest priority.

Lennart

-- 
Lennart Poettering, Red Hat