[systemd-devel] Shutdown order in systemd

Colin Guthrie gmane at colin.guthr.ie
Mon Aug 3 19:01:35 UTC 2020 

Zheng SHAO wrote on 03/08/2020 13:31:
> Hello,
> 
> We are finding a robust way to handle ACPI G2 soft off signal to graceful shutdown our application.
> To simplifier the problem, consider our instance is running with Nginx behind a load balancer.
> When the ACPI G2 soft off signal comes to the Nginx instance, we want to do these jobs
> 
> 1. Keep current HTTP connection works.
> 2. Fail the health check in load balance side.
> 3. Make sure new connection not comes from load balancer.
> 4. Kill long connections if any connection exceeds to 60 seconds.
> 5. Continue shutdown process.
> 
> We are considering to achieve this by 2 options,
> 1. Add a custom handler for `HandlePowerKey` in /etc/systemd/logind.conf.
> 2. Add a system service so when systemd starting shutdown, this service will be run first and block other service to be killed.

Have you considered writing a service that takes a systemd-inhibit
shutdown lock?

This might not work but looking very quickly at
https://www.freedesktop.org/wiki/Software/systemd/inhibit/ it would
appear you get a PrepareForShutdown signal which could kick off your steps.

Depending on how things work, you could just introduce a "delay" inhibit
rather than a "block", e.g. a 70s delay could give you a bit of headroom
to trigger fail the LB's health check.

Or perhaps you could take a block lock, then when the
prepareForShutdown() signal comes in, fail the LB, then when that is
confirmed, add a new 60s delay inhibit (not sure if this works after
shutdown has been triggered), then release the block inhibit and just
wait for everything else to run it's course? Alternatively just keep the
block inhibit right up until you want step 5 to begin.

Again, this is pure speculation and the default handler for
HandlePowerKey may bypass login (tho' I suspect not) and others may
explain other reasons why this approach may not work.

Good luck

Col


> The method[2] is a preferred way, but we can not find a correct implement for this.
> 
> ```
> [Unit]
> Description=Delay shutdown
> After=network-online.target network.target rsyslog.service
> After=google-instance-setup.service google-network-daemon.service
> After=systemd-user-sessions.service sshd.service google-fluentd.service user.slice system.slice nss-user-lookup.target logind.service
> Wants=network-online.target network.target rsyslog.service google-instance-setup.service google-network-daemon.service systemd-user-sessions.service sshd.service google-fluentd.service user.slice system.slice multi-user.target nss-user-lookup.target logind.service
> 
> [Service]
> Type=oneshot
> ExecStart=/bin/true
> ExecStop=/root/shutdown.sh
> RemainAfterExit=yes
> KillMode=none
> TimeoutStopSec=0
> StandardOutput=journal+console
> 
> [Install]
> WantedBy=multi-user.target
> ```
> 
> /root/shutdown.sh
> ```
> #!/bin/bash
> 
> echo start shutdown
> echo sleep 300
> sleep 300
> echo end shutdown
> ```
> 
> We checked console output shows as follow,
> ```
> CentOS Linux 7 (Core)
> Kernel 3.10.0-1127.10.1.el7.x86_64 on an x86_64
> 
> shao-redis-prd-base login: Aug  3 21:19:25 shao-redis-prd-base chronyd[449]: Selected source 169.254.169.254
> Aug  3 21:20:01 shao-redis-prd-base systemd: Created slice User Slice of root.
> Aug  3 21:20:01 shao-redis-prd-base systemd: Started Session 1 of user root.
> Aug  3 21:20:01 shao-redis-prd-base systemd: Removed slice User Slice of root.
> Aug  3 21:20:11 shao-redis-prd-base systemd: Started Unbound recursive Domain Name Server.
> Aug  3 21:20:11 shao-redis-prd-base systemd: Reached target Host and Network Name Lookups.
> Aug  3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init module 0: ipsecmod
> Aug  3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init module 1: validator
> Aug  3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init module 2: iterator
> Aug  3 21:20:11 shao-redis-prd-base unbound: [1204:0] info: start of service (unbound 1.6.6).
> Aug  3 21:20:59 shao-redis-prd-base systemd: Created slice User Slice of xxx.
> Aug  3 21:20:59 shao-redis-prd-base systemd: Started Session 2 of user xxx.
> Aug  3 21:21:29 shao-redis-prd-base systemd: Reached target Multi-User System.
> Aug  3 21:21:29 shao-redis-prd-base systemd: Starting Update UTMP about System Runlevel Changes...
> Aug  3 21:21:29 shao-redis-prd-base systemd: Started Update UTMP about System Runlevel Changes.
> Aug  3 21:21:29 shao-redis-prd-base systemd: Startup finished in 768ms (kernel) + 2.006s (initrd) + 2min 13.532s (userspace) = 2min 16.307s.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Removed slice system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped ACPI Event Daemon.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Authorization Manager.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Job spooling tools.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Getty on tty1.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Serial Getty on ttyS0.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped NTP client/server.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Google OSConfig Agent.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Command Scheduler.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Unbound recursive Domain Name Server.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Removed slice system-serial\x2dgetty.slice.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Starting Show Plymouth Power Off Screen...
> Aug  3 21:23:09 shao-redis-prd-base systemd: Removed slice system-getty.slice.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Session 2 of user kuma.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Removed slice User Slice of kuma.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopping Login Service...
> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Login Service.
> [  236.662420] shutdown.sh[1265]: start shutdown
> [  236.662802] shutdown.sh[1265]: sleep 300
> Aug  3 21:23:09 shao-redis-prd-base shutdown.sh: start shutdown
> Aug  3 21:23:09 shao-redis-prd-base shutdown.sh: sleep 300
> Aug  3 21:23:09 shao-redis-prd-base GCEMetadataScripts[1266]: 2020/08/03 21:23:09 GCEMetadataScripts: Starting shutdown scripts (version 20200706.00).
> Aug  3 21:23:09 shao-redis-prd-base GCEMetadataScripts[1266]: 2020/08/03 21:23:09 GCEMetadataScripts: No shutdown scripts to run.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Google Compute Engine Shutdown Scripts.
> Aug  3 21:23:09 shao-redis-prd-base systemd: Received SIGRTMIN+20 from PID 1276 (plymouthd).
> 
> [  OK  ] Started Show Plymouth Power Off Screen.
> 
> Aug  3 21:23:09 shao-redis-prd-base systemd: Started Show Plymouth Power Off Screen.
> 
> [  OK  ] Stopped Dynamic System Tuning Daemon.
> 
> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Dynamic System Tuning Daemon.
> ```
> 
> It seems other system services are also start shutdown at same time, is there a better to do this?
> 
> Any advice is welcome.
> 
> Thanks,
> 


-- 

Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
  Tribalogic Limited http://www.tribalogic.net/
Open Source:
  Mageia Contributor http://www.mageia.org/
  PulseAudio Hacker http://www.pulseaudio.org/
  Trac Hacker http://trac.edgewall.org/

More information about the systemd-devel mailing list