[systemd-devel] systemd update "forgets" ordering for shutdown

[Michael Chapman]

On Sun, 17 May 2020, Michael Chapman wrote:
> On Fri, 15 May 2020, Frank Steiner wrote:
> > Hi,
> > 
> > I need to run a script on shutdown before any other service is stopped.
> > Due to an advice Lennart gave a while ago I'm using this service file
> > (with multi-user.target being our default runlevel target):
> > 
> > [Unit]
> > After=multi-user.target
> > 
> > [Service]
> > Type=oneshot
> > ExecStart=/bin/true
> > ExecStop=/usr/lib/systemd/scripts/halt.local.bio
> > TimeoutSec=120
> > RemainAfterExit=yes
> 
> This seems inherently fragile.
> 
> If `multi-user.target` were to be stopped for whatever reason (and this 
> is generally possible), the ordering dependencies between services 
> Before=multi-user.target and services After=multi-user.target are broken. 
> This is because no stop job for `multi-user.target` will be added to 
> systemd's transaction (it's already stopped!), and ordering dependencies 
> only apply to units actually added to the transaction.

A quick test can demonstrate this.

First, let's set up a few test units:

    $ systemctl --user cat test-a.service test-b.service start-tests.target stop-tests.target
    # /home/mchapman/.config/systemd/user/test-a.service
    [Unit]
    Conflicts=stop-tests.target

    [Service]
    ExecStart=/bin/echo start a
    ExecStop=/bin/echo stop a
    RemainAfterExit=true

    [Install]
    WantedBy=start-tests.target

    # /home/mchapman/.config/systemd/user/test-b.service
    [Unit]
    Conflicts=stop-tests.target
    After=start-tests.target

    [Service]
    ExecStart=/bin/echo start b
    ExecStop=/bin/echo stop b
    RemainAfterExit=true

    [Install]
    WantedBy=start-tests.target

    # /home/mchapman/.config/systemd/user/start-tests.target
    [Unit]
    Conflicts=stop-tests.target

    # /home/mchapman/.config/systemd/user/stop-tests.target
    [Unit]
    Conflicts=start-tests.target

The idea here is that `start-tests.target` is like `multi-user.target`, 
and `stop-tests.target` is like `shutdown.target` (or one of the other 
targets normally used during shutdown).

`test-a.service` is ordered before `start-tests.target`, but 
`test-b.service` is ordered after it.

Next we enable `test-a.service` and `test-b.service`:

    $ systemctl --user enable test-a.service test-b.service
    Created symlink /home/mchapman/.config/systemd/user/start-tests.target.wants/test-a.service -> /home/mchapman/.config/systemd/user/test-a.service.
    Created symlink /home/mchapman/.config/systemd/user/start-tests.target.wants/test-b.service -> /home/mchapman/.config/systemd/user/test-b.service.

With this in place, both simulated "boot" and "shutdown" work correctly. 
Simulating boot:

    $ systemctl --user start start-tests.target
    $ journalctl --user
    May 17 11:01:07 beren.home systemd[2186]: Started test-a.service.
    May 17 11:01:07 beren.home systemd[2186]: Reached target start-tests.target.
    May 17 11:01:07 beren.home echo[983330]: start a
    May 17 11:01:07 beren.home systemd[2186]: Started test-b.service.
    May 17 11:01:07 beren.home echo[983331]: start b

Simulating shutdown:

    $ systemctl --user start stop-tests.target
    $ journalctl --user
    May 17 11:01:11 beren.home systemd[2186]: Reached target stop-tests.target.
    May 17 11:01:11 beren.home systemd[2186]: Stopping test-b.service...
    May 17 11:01:11 beren.home systemd[2186]: test-b.service: Succeeded.
    May 17 11:01:13 beren.home echo[983355]: stop b
    May 17 11:01:11 beren.home systemd[2186]: Stopped test-b.service.
    May 17 11:01:13 beren.home echo[983359]: stop a
    May 17 11:01:11 beren.home systemd[2186]: Stopped target start-tests.target.
    May 17 11:01:11 beren.home systemd[2186]: Stopping test-a.service...
    May 17 11:01:11 beren.home systemd[2186]: test-a.service: Succeeded.
    May 17 11:01:11 beren.home systemd[2186]: Stopped test-a.service.

But what happens if we were to stop `start-tests.target` in the middle of 
this?

    $ systemctl --user start start-tests.target
    $ journalctl --user
    May 17 11:01:23 beren.home systemd[2186]: Stopped target stop-tests.target.
    May 17 11:01:23 beren.home systemd[2186]: Started test-a.service.
    May 17 11:01:23 beren.home systemd[2186]: Reached target start-tests.target.
    May 17 11:01:23 beren.home echo[983369]: start a
    May 17 11:01:23 beren.home systemd[2186]: Started test-b.service.
    May 17 11:01:23 beren.home echo[983370]: start b
    May 17 11:01:28 beren.home systemd[2186]: Stopped target start-tests.target.

    $ systemctl --user stop start-tests.target
    $ journalctl --user
    May 17 11:01:28 beren.home systemd[2186]: Stopped target start-tests.target.

    $ systemctl --user start stop-tests.target
    $ journalctl --user
    May 17 11:01:34 beren.home systemd[2186]: Reached target stop-tests.target.
    May 17 11:01:34 beren.home systemd[2186]: Stopping test-a.service...
    May 17 11:01:34 beren.home systemd[2186]: Stopping test-b.service...
    May 17 11:01:34 beren.home echo[983397]: stop a
    May 17 11:01:34 beren.home systemd[2186]: test-a.service: Succeeded.
    May 17 11:01:34 beren.home systemd[2186]: Stopped test-a.service.
    May 17 11:01:34 beren.home echo[983398]: stop b
    May 17 11:01:34 beren.home systemd[2186]: test-b.service: Succeeded.
    May 17 11:01:34 beren.home systemd[2186]: Stopped test-b.service.

`test-a.service` was stopped first this time (check the PIDs!). This is 
because the ordering dependency between `test-a.service` and 
`test-b.service` is lost since `start-tests.target` wasn't active.