[systemd-devel] journald deleting logs on LiveOS boots

[Lennart Poettering]

On Di, 18.06.19 20:34, Chris Murphy (lists at colorremedies.com) wrote:

> When I boot Fedora 30/Rawhide Workstation (LiveOS install media) in a
> VM with ~2GiB memory, using any combination of systemd, dracut, or
> udev debugging, events are missing from the journal.
>
> systemd-241-7.gita2eaa1c.fc30.x86_64
> systemd-242-3.git7a6d834.fc31.x86_64
>
> 'journalctl -b -o short-monotonic' starts at ~22s monotonic time. i.e.
> all the messages before that are not in the journal. This problem
> doesn't happen with netinstalls. Fedora LiveOS boots setup a memory
> based persistent overlay, and /var/log/journal exists and

what do you mean by "memory based persistent overlay"? if its in
memory it's not persistant, is it? /me is confused...

is this LVM stuff or overlayfs?

> systemd-journald tries to flush there, whereas on netinstalls
> /var/log/journal does not exist.
>
> Using systemd.log_target=kmsg I discovered that systemd-journald is
> deleting logs in the LiveOS case, but I don't know why, / has ~750M
> free
>
> [   24.910792] systemd-journald[922]: Vacuuming...
> [   24.921802] systemd-journald[922]: Deleted archived journal
> /var/log/journal/05d0a9c86a0e4bbcb36c5e0082b987ee/system at 818f3f40f19849e08a1b37b9c1e304f1-0000000000000001-00058ba31bec725e.journal
> (8.0M).
> [   24.921808] systemd-journald[922]: Vacuuming done, freed 8.0M of
> archived journals from
> /var/log/journal/05d0a9c86a0e4bbcb36c5e0082b987ee.
>
> I filed a bug here and have nominated it as a blocker bug for the
> F31 cycle.

Note that journald logs early on how much disk space it assumes to
have free, you might see that in dmesg? (if not, boot with
systemd.journald.forward_to_kmsg=1 on the kernel cmdline)

> A related question is whether it's even appropriate or
> /var/log/journal to exist on LiveOS boots, rather than just have
> journals go to /run/log/journal? Obviously we must have the entire
> journal for LiveOS boots, from monotonic time 0.0, no matter the debug
> options chosen, I'd say for at least 5 minutes? Almost immediately
> dropping the first 30s just because I've got debug options set is a
> big problem for troubleshooting other problems with early startup of
> install images.

There is not much point in using /var if /run works too and /var isn't
persistent anyway...

Lennart

--
Lennart Poettering, Berlin