[systemd-devel] journald deleting logs on LiveOS boots

[Chris Murphy]

This is still a problem with systemd-242-5.git7a6d834.fc31.x86_64

If I boot using 'systemd.journald.forward_to_console=1
console=ttyS0,38400 console=tty1 systemd.log_level=debug rd.debug
rd.udev.debug'

There is no debug output forwarded to console, only kernel messages
and normal systemd logging is forwarded. And of course this bug means
that those debug messages are lost once the vaccuuming happens.

If I boot using 'systemd.log_level=debug rd.debug rd.udev.debug
systemd.log_target=kmsg log_buf_len=64M printk.devkmsg=on'

For sure a bunch of dracut messages are not being forwarded to kmsg,
none of the rd.live.image debug stuff is listed, so I can't even see
how things are being assembled for live boots, with time stamps, to
see if that stuff might not be ready at the time journald switches
from using /run/log to /var/log. I can't even see in kmsg the journald
switch from /run/log to /var/log. That itself seems like a bug, given
systemd.log_target=kmsg, I'd like to think that should cause an exact
copy to dump to kmsg, of what's going to system.journal but apparently
that's not the case.

So far nothing I've tried gets me access to information that would
give a hint why systemd-journald thinks there's no free space and yet
it still decides to create a single 8MB system journal, which then
almost immediately gets deleted, including all the evidence up to that
point.

For sure sysroot and / are available rw by these points:
<31>[   10.898648] systemd[1]: sysroot.mount: About to execute:
/usr/bin/mount /dev/mapper/live-rw /sysroot
...
<31>[   12.061370] systemctl[879]: Switching root - root: /sysroot; init: n/a


This is the loss of the journal up to this point:
<47>[   24.318297] systemd-journald[905]:
/var/log/journal/05d0a9c86a0e4bbcb36c5e0082b987ee/system.journal:
Allocation limit reached, rotating.
<47>[   24.318315] systemd-journald[905]: Rotating...
<47>[   24.332853] systemd-journald[905]: Reserving 147626 entries in
hash table.
<47>[   24.367396] systemd-journald[905]: Vacuuming...
<47>[   24.389952] systemd-journald[905]: Deleted archived journal
/var/log/journal/05d0a9c86a0e4bbcb36c5e0082b987ee/system at 2f2d06548b5f4c259693b56558cc89c6-0000000000000001-00058dbdb33d1f5e.journal
(8.0M).
<47>[   24.389965] systemd-journald[905]: Vacuuming done, freed 8.0M
of archived journals from
/var/log/journal/05d0a9c86a0e4bbcb36c5e0082b987ee.
<47>[   24.390015] systemd-journald[905]: Journal effective settings
seal=no compress=yes compress_threshold_bytes=512B
<47>[   24.390126] systemd-journald[905]: Retrying write.

Retrying what write and why does it need to retry? What failed?

--
Chris Murphy