[systemd-devel] journal always corrupt

[Vito Caputo]

On Thu, Jun 07, 2018 at 07:32:10PM +0300, Mantas Mikulėnas wrote:
> On Thu, Jun 7, 2018 at 4:21 AM Chris Murphy <lists at colorremedies.com> wrote:
> 
> > [chris at f28h ~]$ sudo journalctl --verify
> > 15f1c8: Data object references invalid entry at 4855f8
> > File corruption detected at
> > /run/log/journal/bbe68372db9f4c589a1f67f008e70864/system.journal:4854c0
> > (of 8388608 bytes, 56%).
> > FAIL: /run/log/journal/bbe68372db9f4c589a1f67f008e70864/system.journal
> > (Bad message)
> > PASS: /var/log/journal/bbe68372db9f4c589a1f67f008e70864/system.journal
> > PASS: /var/log/journal/bbe68372db9f4c589a1f67f008e70864/user-1000.journal
> > [chris at f28h ~]$ ls -l /run/log/journal/bbe68372db9f4c589a1f67f008e70864/
> > total 8192
> > -rw-r-----+ 1 root systemd-journal 8388608 Jun  6 14:28 system.journal
> > [chris at f28h ~]$
> >
> > systemd-238-8.git0e0aa59.fc28.x86_64
> >
> > It doesn't seem to matter whether this is on volatile or persistent
> > media, the very first journal file has corruption, subsequent ones
> > don't. I'm not sure how to troubleshoot this.
> >
> 
> More precisely, it's the *active* journal file, the one that journald is
> currently writing to. If it has been just a few seconds since the last
> write, you can probably safely assume that it's not fully flushed to disk
> yet. (This can apply to user-* journals as well, but they're relatively low
> traffic and so less likely to be online at the moment.)
> 
> I would use `journalctl --rotate` to make journald start a new file, so
> that the old one is properly taken offline.
> 

This doesn't make sense to me.  Flushing to disk is relevant to
crash/reboot durability, but not to other processes reading from the
file while the system and journald are operating normally.  The journal
writing is done via MAP_SHARED mmap windows in journald, if something
reads from the journals, it will see the current file state from the
perpective of the kernel's VM subsystem, not necessarily the on-disk
state.

I don't know why Chris is experiencing what he's seeing, it's not
something I have time to look more closely at right now.  At a glance
though it seems suspect and worth investigating.

Regards,
Vito Caputo