[systemd-devel] journald deleting logs on LiveOS boots

[Chris Murphy]

When I boot Fedora 30/Rawhide Workstation (LiveOS install media) in a
VM with ~2GiB memory, using any combination of systemd, dracut, or
udev debugging, events are missing from the journal.

systemd-241-7.gita2eaa1c.fc30.x86_64
systemd-242-3.git7a6d834.fc31.x86_64

'journalctl -b -o short-monotonic' starts at ~22s monotonic time. i.e.
all the messages before that are not in the journal. This problem
doesn't happen with netinstalls. Fedora LiveOS boots setup a memory
based persistent overlay, and /var/log/journal exists and
systemd-journald tries to flush there, whereas on netinstalls
/var/log/journal does not exist.

Using systemd.log_target=kmsg I discovered that systemd-journald is
deleting logs in the LiveOS case, but I don't know why, / has ~750M
free

[   24.910792] systemd-journald[922]: Vacuuming...
[   24.921802] systemd-journald[922]: Deleted archived journal
/var/log/journal/05d0a9c86a0e4bbcb36c5e0082b987ee/system at 818f3f40f19849e08a1b37b9c1e304f1-0000000000000001-00058ba31bec725e.journal
(8.0M).
[   24.921808] systemd-journald[922]: Vacuuming done, freed 8.0M of
archived journals from
/var/log/journal/05d0a9c86a0e4bbcb36c5e0082b987ee.

I filed a bug here and have nominated it as a blocker bug for the F31 cycle.

LiveOS boot, journalctl is missing many early messages
https://bugzilla.redhat.com/show_bug.cgi?id=1715699

Comments 8 and 9 most relevent at this point (rather noisy
troubleshooting before that).

A related question is whether it's even appropriate or
/var/log/journal to exist on LiveOS boots, rather than just have
journals go to /run/log/journal? Obviously we must have the entire
journal for LiveOS boots, from monotonic time 0.0, no matter the debug
options chosen, I'd say for at least 5 minutes? Almost immediately
dropping the first 30s just because I've got debug options set is a
big problem for troubleshooting other problems with early startup of
install images.

Thanks,


-- 
Chris Murphy