[systemd-devel] Journal file is synced each minute into disk

[Ricardo Fraile]

No, there aren't any new messages in that time. This example was
reproduced in a testing server without load, but in other server with
nearly 10Mb of logs each minute the behaviour is the same.

Following the way to the root cause I fall into a high kworker I/O usage
each minute, some related with ext4 journal said the following command:

while true ; do clear ; grep -n ^ /proc/$kworker_PID/stack | sort -rn |
cut -d: -f 2- ; sleep 1 ; done

A search in google reported some similar issues with this filesystem,
but a kernel update from 3.16.7 to 4.5.0 didn't fix anything.

Only removing the journal support in the ext4 log partition solved it,
but I still don't understand why if SyncIntervalSec sync each 5 minutes,
for example, the journal is so aggressive each minute in a server
without disk writes, apart of logs.

Anyway, the problem is close to the filesystem side, I don't know if any
other options defined in journald.conf can mitigate it.


Maybe this message can help another with this issue.


Thanks,



El lun, 09-05-2016 a las 22:02 +0300, Andrei Borzenkov escribió:
> 09.05.2016 18:21, Ricardo Fraile пишет:
> > Hello,
> > 
> > 
> > With the following configuration in "/etc/systemd/journald.conf":
> > 
> > [Journal]
> > Seal=no
> > SplitMode=none
> > SyncIntervalSec=5m
> > RateLimitInterval=0
> > RateLimitBurst=0
> > ForwardToSyslog=no
> > 
> > SystemMaxUse=5G
> > SystemKeepFree=500M
> > 
> > RuntimeMaxUse=500M
> > RuntimeKeepFree=250M
> > 
> > 
> > 
> > The file "system.journal" located under "/var/log/journal/[number]/" is
> > synced each minute, for example:
> > 
> > # ls -al
> > -rw-r----- 1 root root 8388608 May  9 17:05 system.journal
> > 
> > # ls -al
> > -rw-r----- 1 root root 8388608 May  9 17:06 system.journal
> > 
> > 
> 
> It has nothing to do with sync to disk; it simply shows file
> modification time which is kept in memory as well. Are there any new
> messages in this minute?
> 
> > 
> > But its extrange since the variable "SyncIntervalSec" have the "5m"
> > value and the man page said:
> > 
> > SyncIntervalSec=
> > The timeout before synchronizing journal files to disk. After syncing,
> > journal files are placed in the OFFLINE state. Note that syncing is
> > unconditionally done immediately after a log message of priority CRIT,
> > ALERT or EMERG has been logged. This setting hence applies only to
> > messages of the levels ERR, WARNING, NOTICE, INFO, DEBUG. The default
> > timeout is 5 minutes.
> > 
> > 
> > Ok, maybe any important error is generated, but is not the case, the
> > journal didn't report anything:
> > 
> > # journalctl -p crit -p alert -p emerg
> > -- Logs begin at Mon 2016-05-09 15:51:12 CEST, end at Mon 2016-05-09
> > 17:12:54 CEST. --
> > 
> > 
> > Is this the right behaviour?
> > 
> > How is possible to delay the sync of the persistent journal file?
> > 
> > 
> > # systemd --version
> > systemd 215
> > +PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ
> > -SECCOMP -APPARMOR
> > ...runing in Debian Jessie 8.4
> > 
> > Thanks,
> > 
> > _______________________________________________
> > systemd-devel mailing list
> > systemd-devel at lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/systemd-devel
> > 
>