[systemd-devel] Shutdown order in systemd

[Zheng SHAO]

Hello,

We are finding a robust way to handle ACPI G2 soft off signal to graceful shutdown our application.
To simplifier the problem, consider our instance is running with Nginx behind a load balancer.
When the ACPI G2 soft off signal comes to the Nginx instance, we want to do these jobs

1. Keep current HTTP connection works.
2. Fail the health check in load balance side.
3. Make sure new connection not comes from load balancer.
4. Kill long connections if any connection exceeds to 60 seconds.
5. Continue shutdown process.

We are considering to achieve this by 2 options,
1. Add a custom handler for `HandlePowerKey` in /etc/systemd/logind.conf.
2. Add a system service so when systemd starting shutdown, this service will be run first and block other service to be killed.

The method[2] is a preferred way, but we can not find a correct implement for this.

```
[Unit]
Description=Delay shutdown
After=network-online.target network.target rsyslog.service
After=google-instance-setup.service google-network-daemon.service
After=systemd-user-sessions.service sshd.service google-fluentd.service user.slice system.slice nss-user-lookup.target logind.service
Wants=network-online.target network.target rsyslog.service google-instance-setup.service google-network-daemon.service systemd-user-sessions.service sshd.service google-fluentd.service user.slice system.slice multi-user.target nss-user-lookup.target logind.service

[Service]
Type=oneshot
ExecStart=/bin/true
ExecStop=/root/shutdown.sh
RemainAfterExit=yes
KillMode=none
TimeoutStopSec=0
StandardOutput=journal+console

[Install]
WantedBy=multi-user.target
```

/root/shutdown.sh
```
#!/bin/bash

echo start shutdown
echo sleep 300
sleep 300
echo end shutdown
```

We checked console output shows as follow,
```
CentOS Linux 7 (Core)
Kernel 3.10.0-1127.10.1.el7.x86_64 on an x86_64

shao-redis-prd-base login: Aug  3 21:19:25 shao-redis-prd-base chronyd[449]: Selected source 169.254.169.254
Aug  3 21:20:01 shao-redis-prd-base systemd: Created slice User Slice of root.
Aug  3 21:20:01 shao-redis-prd-base systemd: Started Session 1 of user root.
Aug  3 21:20:01 shao-redis-prd-base systemd: Removed slice User Slice of root.
Aug  3 21:20:11 shao-redis-prd-base systemd: Started Unbound recursive Domain Name Server.
Aug  3 21:20:11 shao-redis-prd-base systemd: Reached target Host and Network Name Lookups.
Aug  3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init module 0: ipsecmod
Aug  3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init module 1: validator
Aug  3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init module 2: iterator
Aug  3 21:20:11 shao-redis-prd-base unbound: [1204:0] info: start of service (unbound 1.6.6).
Aug  3 21:20:59 shao-redis-prd-base systemd: Created slice User Slice of xxx.
Aug  3 21:20:59 shao-redis-prd-base systemd: Started Session 2 of user xxx.
Aug  3 21:21:29 shao-redis-prd-base systemd: Reached target Multi-User System.
Aug  3 21:21:29 shao-redis-prd-base systemd: Starting Update UTMP about System Runlevel Changes...
Aug  3 21:21:29 shao-redis-prd-base systemd: Started Update UTMP about System Runlevel Changes.
Aug  3 21:21:29 shao-redis-prd-base systemd: Startup finished in 768ms (kernel) + 2.006s (initrd) + 2min 13.532s (userspace) = 2min 16.307s.
Aug  3 21:23:09 shao-redis-prd-base systemd: Removed slice system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice.
Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped ACPI Event Daemon.
Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Authorization Manager.
Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Job spooling tools.
Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Getty on tty1.
Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Serial Getty on ttyS0.
Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped NTP client/server.
Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Google OSConfig Agent.
Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Command Scheduler.
Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Unbound recursive Domain Name Server.
Aug  3 21:23:09 shao-redis-prd-base systemd: Removed slice system-serial\x2dgetty.slice.
Aug  3 21:23:09 shao-redis-prd-base systemd: Starting Show Plymouth Power Off Screen...
Aug  3 21:23:09 shao-redis-prd-base systemd: Removed slice system-getty.slice.
Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Session 2 of user kuma.
Aug  3 21:23:09 shao-redis-prd-base systemd: Removed slice User Slice of kuma.
Aug  3 21:23:09 shao-redis-prd-base systemd: Stopping Login Service...
Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Login Service.
[  236.662420] shutdown.sh[1265]: start shutdown
[  236.662802] shutdown.sh[1265]: sleep 300
Aug  3 21:23:09 shao-redis-prd-base shutdown.sh: start shutdown
Aug  3 21:23:09 shao-redis-prd-base shutdown.sh: sleep 300
Aug  3 21:23:09 shao-redis-prd-base GCEMetadataScripts[1266]: 2020/08/03 21:23:09 GCEMetadataScripts: Starting shutdown scripts (version 20200706.00).
Aug  3 21:23:09 shao-redis-prd-base GCEMetadataScripts[1266]: 2020/08/03 21:23:09 GCEMetadataScripts: No shutdown scripts to run.
Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Google Compute Engine Shutdown Scripts.
Aug  3 21:23:09 shao-redis-prd-base systemd: Received SIGRTMIN+20 from PID 1276 (plymouthd).

[  OK  ] Started Show Plymouth Power Off Screen.

Aug  3 21:23:09 shao-redis-prd-base systemd: Started Show Plymouth Power Off Screen.

[  OK  ] Stopped Dynamic System Tuning Daemon.

Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Dynamic System Tuning Daemon.
```

It seems other system services are also start shutdown at same time, is there a better to do this?

Any advice is welcome.

Thanks,

-- 
Zheng Shao