[systemd-devel] journald deleting logs on LiveOS boots

[Chris Murphy]

On Thu, Jul 18, 2019 at 4:02 PM Chris Murphy <lists at colorremedies.com> wrote:
>
> On Thu, Jul 18, 2019 at 10:18 AM Dave Howorth <systemd at howorth.org.uk> wrote:
> >
> > On Thu, 18 Jul 2019 09:55:51 -0600
> > Chris Murphy <lists at colorremedies.com> wrote:
> > > On Thu, Jul 18, 2019 at 4:50 AM Uoti Urpala <uoti.urpala at pp1.inet.fi>
> > > wrote:
> > > >
> > > > On Mon, 2019-07-15 at 14:32 -0600, Chris Murphy wrote:
> > > > > So far nothing I've tried gets me access to information that would
> > > > > give a hint why systemd-journald thinks there's no free space and
> > > > > yet it still decides to create a single 8MB system journal, which
> > > > > then almost immediately gets deleted, including all the evidence
> > > > > up to that point.
> > > >
> > > > Run journald under strace and check the results of the system calls
> > > > used to query space? (One way to run it under strace would be to
> > > > change the unit file to use "strace -D -o /run/output
> > > > systemd-journald" as the process to start.)
> > >
> > > It's a good idea but strace isn't available on Fedora live media. So I
> > > either have to learn how to create a custom live media locally (it's a
> > > really complicated process) or convince Fedora to add strace to live
> > > media...
> >
> > I'm not a fedora user, but I don't think it's that difficult to run
> > strace.
> >
> > To run it once, start your live image and type:
> >
> > # yum install strace
> >
> > You will need to reinstall it if you reboot.
> >
> > To permanently install it apparently you need to configure your USB
> > with persistent storage. I haven't looked up how to do that.
>
> I thought about that, but this is a substantial alteration from the
> original ISO in terms of the storage layout and how everything gets
> assembled. But it's worth a shot. If it is a systemd bug, then it
> should still reproduce. If it doesn't reproduce, then chances are it's
> some kind of assembly related problem.
>
> Still seems like a systemd-journald bug that neither forward to
> console nor to kmsg includes any useful systemd or dracut debugging.

I used livecd-iso-to-disk to create the persistent boot media, the
problem does reproduce. But I needed a custom initramfs that has the
modified systemd-journald.service unit file in it, as well as strace,
because this problem happens that early in the startup process. Thing
is, that custom initramfs blows up trying to assemble the persistent
overlay. Looking at the compose logs for this image, I see a rather
complex dracut command is needed:

dracut --nomdadmconf --nolvmconf --xz --add livenet dmsquash-live
convertfs pollcdrom qemu qemu-net --omit plymouth --no-hostonly
--debug --no-early-microcode --force
boot/initramfs-5.3.0-0.rc0.git4.1.fc31.x86_64.img
5.3.0-0.rc0.git4.1.fc31.x86_64

Fine. Now I can boot. Next gotcha is that /run/output is a 0 byte
length file. I don't know why. I've booted with enforcing=0 in case
selinux doesn't like strace writing to /run/ but that hasn't made a
difference.

The change I made in /usr/lib/systemd/system/systemd-journald.service is
ExecStart=/usr/bin/strace -D -o /run/output /usr/lib/systemd/systemd-journald

I've also tried booting with the original initrd which doesn't have
the systemd-journald modification, relying only on the one on
persistent storage. I still get /run/output as a 0 length file. I've
tried booting with enforcing=0 in case selinux is denying the write,
but that doesn't solve the problem.

So I'm still stuck.


-- 
Chris Murphy