[systemd-devel] Generic unit file to run command on shutdown impossible?

[David Mazieres expires 2013-11-08 PST]

I'm trying to do something that I hope is relatively simple, namely to
run a simple kexec -l ... command on system shutdown before the /boot
file system is unmounted.  I'm trying this on two different machines,
both of which are running current versions of arch linux with systemd
204.  The machines are configured very similarly, except that one has
a serial console and some NFS mounts, while the other does not.
Unfortunately, I cannot find a single script that works on both
machines.

On the machine with a serial console, the following unit file seems to
work (in /etc/systemd/system/kexec-load at .service, enabled with
systemctl enable kexec-load at linux).  This is a slightly modified
version of a suggestion on the arch linux wiki:

	[Unit]
	Description=load %i kernel into the current kernel
	Documentation=man:kexec(8)
	DefaultDependencies=no
	Before=shutdown.target umount.target final.target
	After=sysinit.target

	[Service]
	Type=oneshot
	ExecStart=/usr/bin/kexec -l /boot/vmlinuz-%i --initrd=/boot/initramfs-%i.img --reuse-cmdline

	[Install]
	WantedBy=kexec.target

However, when I try this script on the machine without a serial
console, the script gets run at the very end of the shutdown process,
after all the file systems are unmounted (and hence at a point where
the new kernel image is unavailable).  I can boot with
systemd.log_level=debug, but since there's no serial console and I
can't get my job to fire before the journal system is shutdown, it's
hard for me to capture all the output on shutdown.

Conversely, I devised a kind of hacky way to make it work on the
non-serial-console machine what doesn't work on the serial console
one.  Here my idea is to have a job whose ExecStop loads the new
kernel.  I'm not sure why that job fires only on startup and not
shutdown of the serial console machine, but if I solve one of these
problems, I'd rather get the WantedBy=kexec.target approach working,
if that's possible, so I'll omit the details for now.

For what it's worth, here's a relevant subset of the output of the
systemd --test command on the machine that does not work (i.e., the
non-serial console) with the above unit file:

	/usr/lib/systemd/systemd --system --test --unit=kexec.target

	-> By units:
		-> Unit shutdown.target:
			After: kexec-load at linux.service
			ReferencedBy: kexec-load at linux.service
		-> Unit systemd-journald.socket:
			Before: kexec-load at linux.service
			ReferencedBy: kexec-load at linux.service
		-> Unit sysinit.target:
			Before: kexec-load at linux.service
			ReferencedBy: kexec-load at linux.service
		-> Unit final.target:
			After: kexec-load at linux.service
			ReferencedBy: kexec-load at linux.service
		-> Unit umount.target:
			After: kexec-load at linux.service
			ReferencedBy: kexec-load at linux.service
		-> Unit kexec.target:
			Wants: kexec-load at linux.service
			References: kexec-load at linux.service
		-> Unit kexec-load at linux.service:
			Name: kexec-load at linux.service
			Fragment Path: /etc/systemd/system/kexec-load at .service
			-> ExecStart:
				Command Line: /usr/bin/kexec -l /boot/vmlinuz-%i --initrd=/boot/initramfs-%i.img --reuse-cmdline

So here the dependencies seem fine.  Yet when I look at the "by jobs"
category:

	-> By jobs:
		-> Job 3:
			Action: kexec.target -> start
			State: waiting
			Forced: no
			Irreversible: no
		-> Job 4:
			Action: systemd-kexec.service -> start
			State: waiting
			Forced: no
			Irreversible: no
		-> Job 5:
			Action: shutdown.target -> start
			State: waiting
			Forced: no
			Irreversible: no
		-> Job 83:
			Action: umount.target -> start
			State: waiting
			Forced: no
			Irreversible: no
		-> Job 88:
			Action: boot.mount -> stop
			State: waiting
			Forced: no
			Irreversible: no
		-> Job 90:
			Action: final.target -> start
			State: waiting
			Forced: no
			Irreversible: no
		-> Job 91:
			Action: kexec-load at linux.service -> start
			State: waiting
			Forced: no
			Irreversible: no
	[end of file]

Job 91 is the very last job.  So while I'm not 100% sure how to parse
this output, it sure looks as though kexec-load is getting started
*after* boot.mount is stopped, which is indeed the behavior I see when
rebooting my machine with "systemctl kexec".  On the other hand, maybe
"waiting" means these jobs are waiting for dependencies?

At any rate, I'd be grateful for any advice on what the problem is, or
even better for any meta-advice on how to debug these things in
general.  (In a sense, debugging my dependencies and solving the
problem are the same thing, as getting my shutdown script to run while
the journal service is active would solve the problem.)

Also, in order to short-circuit some potentially redundant discussion,
I should say that I've also tried adding various permutations of the
following to my unit file, all to no avail:

	RequiresMountsFor=/boot
	RequiresMountsFor=/boot/vmlinuz-linux
	Requires=boot.mount
	After=boot.mount

Thanks.