[systemd-devel] Unit shutdown order not always respected

[David Gubler]

Hi list,

I have a situation where I need to run a command and wait for its 
completion before unmounting a file system (/enc in my case). My problem 
is that systemd sometimes waits for the completion of the command, and 
sometimes doesn't.

So the setup is:
* /enc is a mounted encrypted (luks) volume
* /var/lib/mysql is a bind mount to /enc/mysql
* MariaDB is using /var/lib/mysql
* We've set up a "requires" and "after" dependency chain from MariaDB 
all the way to the luks volume. This works 100% reliably during startup, 
even if something takes too long, fails, or if we have to manually 
fiddle with stuff.
* Ubuntu 20.04 with systemd 245.4


The unit of the command that needs to run before unmounting /enc looks 
like this:


[Unit]
Description=server-secrets-prepare-reboot-enc service
After=network-online.target enc.mount
Requires=network-online.target
BindsTo=enc.mount

[Service]
ExecStart=/bin/true
ExecStop=/usr/local/sbin/server-secrets reboot "/enc"
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=server-secrets-prepare-reboot-enc
User=root
Type=oneshot
RemainAfterExit=yes

[Install]
WantedBy=enc.mount




I've added a 10s sleep to the "server-secrets" command in order to 
eliminate "works by chance" situations.

In normal circumstances everything works perfectly. I can reboot the 
server and systemd waits >10s with unmounting /enc until server-secrets 
is done. I can stop mariadb, umount /var/lib/mysql and /enc, and systemd 
runs server-secrets just fine. I can re-mount /enc, unmount it again, 
everything works.

Where things start to fall apart is when I reboot the server ("reboot" 
or "shutdown -r now") after manually stopping MariaDB. In this case 
systemd starts the server-secrets command before unmounting /enc, but 
does not wait for its completion, and immediately unmounts /enc, causing 
the command to fail.

Note that manually stopping MariaDB does not unmount anything, it 
just... stops MariaDB. But somehow it causes systemd to change its 
ordering behavior during shutdown.

Does this ring any bells?

Thank you!

Best regards,

David Gubler



-- 
David Gubler
System Engineer

VSHN AG | Neugasse 10 | CH-8005 Zürich
+41 44 545 53 00 | https://vshn.ch