[systemd-devel] Shutdown order in systemd

Colin Guthrie gmane at colin.guthr.ie
Tue Aug 11 16:23:02 UTC 2020 

Just FYI and for the sake of cross referencing, the inhibition logic was
mentioned on the list today in a thread: "systemd-inhibit don't work".

A developer says he will work on the patch for this RFE shortly.

Col

Zheng SHAO wrote on 04/08/2020 13:39:
> Hello,
>
> First thanks for your advise.
>
> I didn’t know inhibit before, today I read the document
> and did few simple tests, here is one of these.
>
> $ sudo systemd-inhibit --what=shutdown --who=graceful-shutdown
> --why="Keep application working" --mode=“block” /bin/sleep 60
>
> Unfortunately once ACPI G2 soft off signal comes, the system begin to
> shutdown immediately, I’m still figuring out why systemd-inhibit did
> not block the shutdown process.
>
> At the same time, I found an interesting project which try to block
> shutdown completely.
> https://github.com/ryran/reboot-guard
>
> Thanks!
>
>> On Aug 4, 2020, at 4:01, Colin Guthrie <gmane at colin.guthr.ie
>> <mailto:gmane at colin.guthr.ie>> wrote:
>>
>> Zheng SHAO wrote on 03/08/2020 13:31:
>>> Hello,
>>>
>>> We are finding a robust way to handle ACPI G2 soft off signal to
>>> graceful shutdown our application.
>>> To simplifier the problem, consider our instance is running with
>>> Nginx behind a load balancer.
>>> When the ACPI G2 soft off signal comes to the Nginx instance, we
>>> want to do these jobs
>>>
>>> 1. Keep current HTTP connection works.
>>> 2. Fail the health check in load balance side.
>>> 3. Make sure new connection not comes from load balancer.
>>> 4. Kill long connections if any connection exceeds to 60 seconds.
>>> 5. Continue shutdown process.
>>>
>>> We are considering to achieve this by 2 options,
>>> 1. Add a custom handler for `HandlePowerKey` in
>>> /etc/systemd/logind.conf.
>>> 2. Add a system service so when systemd starting shutdown, this
>>> service will be run first and block other service to be killed.
>>
>> Have you considered writing a service that takes a systemd-inhibit
>> shutdown lock?
>>
>> This might not work but looking very quickly at
>> https://www.freedesktop.org/wiki/Software/systemd/inhibit/ it would
>> appear you get a PrepareForShutdown signal which could kick off your
>> steps.
>>
>> Depending on how things work, you could just introduce a "delay" inhibit
>> rather than a "block", e.g. a 70s delay could give you a bit of headroom
>> to trigger fail the LB's health check.
>>
>> Or perhaps you could take a block lock, then when the
>> prepareForShutdown() signal comes in, fail the LB, then when that is
>> confirmed, add a new 60s delay inhibit (not sure if this works after
>> shutdown has been triggered), then release the block inhibit and just
>> wait for everything else to run it's course? Alternatively just keep the
>> block inhibit right up until you want step 5 to begin.
>>
>> Again, this is pure speculation and the default handler for
>> HandlePowerKey may bypass login (tho' I suspect not) and others may
>> explain other reasons why this approach may not work.
>>
>> Good luck
>>
>> Col
>>
>>
>>> The method[2] is a preferred way, but we can not find a correct
>>> implement for this.
>>>
>>> ```
>>> [Unit]
>>> Description=Delay shutdown
>>> After=network-online.target network.target rsyslog.service
>>> After=google-instance-setup.service google-network-daemon.service
>>> After=systemd-user-sessions.service sshd.service
>>> google-fluentd.service user.slice system.slice
>>> nss-user-lookup.target logind.service
>>> Wants=network-online.target network.target rsyslog.service
>>> google-instance-setup.service google-network-daemon.service
>>> systemd-user-sessions.service sshd.service google-fluentd.service
>>> user.slice system.slice multi-user.target nss-user-lookup.target
>>> logind.service
>>>
>>> [Service]
>>> Type=oneshot
>>> ExecStart=/bin/true
>>> ExecStop=/root/shutdown.sh
>>> RemainAfterExit=yes
>>> KillMode=none
>>> TimeoutStopSec=0
>>> StandardOutput=journal+console
>>>
>>> [Install]
>>> WantedBy=multi-user.target
>>> ```
>>>
>>> /root/shutdown.sh
>>> ```
>>> #!/bin/bash
>>>
>>> echo start shutdown
>>> echo sleep 300
>>> sleep 300
>>> echo end shutdown
>>> ```
>>>
>>> We checked console output shows as follow,
>>> ```
>>> CentOS Linux 7 (Core)
>>> Kernel 3.10.0-1127.10.1.el7.x86_64 on an x86_64
>>>
>>> shao-redis-prd-base login: Aug  3 21:19:25 shao-redis-prd-base
>>> chronyd[449]: Selected source 169.254.169.254
>>> Aug  3 21:20:01 shao-redis-prd-base systemd: Created slice User
>>> Slice of root.
>>> Aug  3 21:20:01 shao-redis-prd-base systemd: Started Session 1 of
>>> user root.
>>> Aug  3 21:20:01 shao-redis-prd-base systemd: Removed slice User
>>> Slice of root.
>>> Aug  3 21:20:11 shao-redis-prd-base systemd: Started Unbound
>>> recursive Domain Name Server.
>>> Aug  3 21:20:11 shao-redis-prd-base systemd: Reached target Host and
>>> Network Name Lookups.
>>> Aug  3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init
>>> module 0: ipsecmod
>>> Aug  3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init
>>> module 1: validator
>>> Aug  3 21:20:11 shao-redis-prd-base unbound: [1204:0] notice: init
>>> module 2: iterator
>>> Aug  3 21:20:11 shao-redis-prd-base unbound: [1204:0] info: start of
>>> service (unbound 1.6.6).
>>> Aug  3 21:20:59 shao-redis-prd-base systemd: Created slice User
>>> Slice of xxx.
>>> Aug  3 21:20:59 shao-redis-prd-base systemd: Started Session 2 of
>>> user xxx.
>>> Aug  3 21:21:29 shao-redis-prd-base systemd: Reached target
>>> Multi-User System.
>>> Aug  3 21:21:29 shao-redis-prd-base systemd: Starting Update UTMP
>>> about System Runlevel Changes...
>>> Aug  3 21:21:29 shao-redis-prd-base systemd: Started Update UTMP
>>> about System Runlevel Changes.
>>> Aug  3 21:21:29 shao-redis-prd-base systemd: Startup finished in
>>> 768ms (kernel) + 2.006s (initrd) + 2min 13.532s (userspace) = 2min
>>> 16.307s.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Removed slice
>>> system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Aug  3 21:23:09
>>> shao-redis-prd-base systemd: Stopped ACPI Event Daemon.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Authorization
>>> Manager.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Job spooling tools.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Getty on tty1.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Serial Getty on
>>> ttyS0.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped NTP client/server.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Google OSConfig
>>> Agent.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Command Scheduler.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Unbound
>>> recursive Domain Name Server.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Removed slice
>>> system-serial\x2dgetty.slice.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Starting Show Plymouth
>>> Power Off Screen...
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Removed slice
>>> system-getty.slice.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Session 2 of
>>> user kuma.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Removed slice User
>>> Slice of kuma.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopping Login Service...
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Login Service.
>>> [  236.662420] shutdown.sh[1265]: start shutdown
>>> [  236.662802] shutdown.sh[1265]: sleep 300
>>> Aug  3 21:23:09 shao-redis-prd-base shutdown.sh: start shutdown
>>> Aug  3 21:23:09 shao-redis-prd-base shutdown.sh: sleep 300
>>> Aug  3 21:23:09 shao-redis-prd-base GCEMetadataScripts[1266]:
>>> 2020/08/03 21:23:09 GCEMetadataScripts: Starting shutdown scripts
>>> (version 20200706.00).
>>> Aug  3 21:23:09 shao-redis-prd-base GCEMetadataScripts[1266]:
>>> 2020/08/03 21:23:09 GCEMetadataScripts: No shutdown scripts to run.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Google Compute
>>> Engine Shutdown Scripts.
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Received SIGRTMIN+20
>>> from PID 1276 (plymouthd).
>>>
>>> [  OK  ] Started Show Plymouth Power Off Screen.
>>>
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Started Show Plymouth
>>> Power Off Screen.
>>>
>>> [  OK  ] Stopped Dynamic System Tuning Daemon.
>>>
>>> Aug  3 21:23:09 shao-redis-prd-base systemd: Stopped Dynamic System
>>> Tuning Daemon.
>>> ```
>>>
>>> It seems other system services are also start shutdown at same time,
>>> is there a better to do this?
>>>
>>> Any advice is welcome.
>>>
>>> Thanks,
>>>
>>
>>
>> -- 
>>
>> Colin Guthrie
>> gmane(at)colin.guthr.ie <http://colin.guthr.ie/>
>> http://colin.guthr.ie/
>>
>> Day Job:
>>  Tribalogic Limited http://www.tribalogic.net/
>> Open Source:
>>  Mageia Contributor http://www.mageia.org/
>>  PulseAudio Hacker http://www.pulseaudio.org/
>>  Trac Hacker http://trac.edgewall.org/
>

-- 

Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
  Tribalogic Limited http://www.tribalogic.net/
Open Source:
  Mageia Contributor http://www.mageia.org/
  PulseAudio Hacker http://www.pulseaudio.org/
  Trac Hacker http://trac.edgewall.org/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20200811/71b638db/attachment-0001.htm>

More information about the systemd-devel mailing list