[systemd-devel] how to put max time limit on reboot time

[Lennart Poettering]

On Do, 05.04.18 11:56, prashantkumar dhotre (prashantkumardhotre at gmail.com) wrote:

> Hi
> Currently it appears that systemd does not provide a way to put max time
> limit for system reboot.
> 
> So there is no way to get system reboot within specified time limit even if
> hardware watchdog is present in system.
> 
> 
> Using RunTimeWatchDogSec,ShutDownTimeWatchDogSec and
> JobTimeoutSec,JobTimeoutAction= in shutdown.target will not solve this
> issue.
> 
> RunTimeWatchDogSec  configures hardware watchdog (hw wd )with the given
> timeout value and system manager will pat the wd at least once in half of
> that time.
> This wd patting will be done in 1st phase of shutdown also.
> example: set this to 2 min and hw wd gets configured with 2 min timeout and
> system manager keeps patting every 1min.
> 
> ShutDownTimeWatchDogSec will configured HW WD with given value and pats the
> WD during 2nd phase of shutdown.
> 2nd phase of shutdown contains loop of 'try to clean up remaining things'.
> 
> So for ex, if we set ShutDownTimeWatchDogSec to 10 min, then during 2nd
> phase of shutdown , if any task of 'try to clean up remaining things' in
> the loop takes more than 10 min, then only HW WD get fired and system
> reboots.
> 
> So this puts max time out on each iteration of loop and not on entire loop
> 
> So this setting does not help in putting max time out for 2nd phase of
> shutdown.
> 
> But intention of ShutDownTimeWatchDogSec  from man page is : “It works as a
> safety net to ensure that the reboot takes place even if a clean reboot
> attempt times out.”
> 
> Clearly this intention is not served by current implementation of
> ShutDownTimeWatchDogSec.
> 
> 
> 
> Do we need to :
> 
> redefine ShutDownTimeWatchDogSec and use it to put max time out on entire
> 2nd part of shutdown.

That's not watch a watchdog is supposed to do. The "watchdog" concept
is generally used to refer to "hang" checks, i.e. to detect cases
where some program is "stuck", i.e. doesn't respond regularly
enough. It's not supposed to put a limit on the overall runtime of a
program, as long as the program keeps pinging the watchdog.

What precisely are you trying to do? Do you want to set a hard time
limit over the entire shutdown logic as a whole? We currently have no
concept for that, and adding that would make sense but would be
completely new code. We do have an over-all timeout on the first phase
of the shutdown, and we have watchdog concepts for all parts of
shutdown, but neither an over-all timeout on the second phase of the
boot, nor an over-all timeout on the combination of both phases.

Quite frankly though I am not entirely sure how even to implement such
an over-all timeout in a fully robust way, even though I am
sympathetic to adding that.

One option might be fork off a "timebomb" subprocess from logind or
PID1 (probably the latter) when we start shutting down. This timebomb
process would just wait for the timeout to elapse, and when it does
would issue reboot(), and little else. This sounds simple but is
quite complex IRL afaics, for at least two reasons:

1. If the timebomb process is supposed to survive from the first into
   the second phase of the reboot, then it would need to exclude
   itself from the process killing spree the second phase does
   initially. There are concepts for that (specifically, the
   argv[0][0] = '@' hack), but it needs special care to be taken.

2. it would have to release references to any kind of resource that
   might interfere with the shutdown logic (i.e. to the umount spree
   for example). One such resource are file systems appearing in its
   mount table or being pinned by the mmaped executable its running
   from. This is particularly nasty, as this means the binary
   implementing the timebomb would have to copy itself to some tmpfs
   or so, reexec itself there, then change its fs namespace and ensure
   the only remaining mount in it is that tmpfs and everything else is
   released. That's doable, but jeeez, far from trivial to
   implement...

Not doing #2 would mean some fs might stay mounted until the last
phase of boot, i.e. would cause it to stay possibly dirty, and if you
care about reliable, clean boots, then that's the worst option.

So yeah, if this is what you are asking for, then please file an RFE
about it. But given the complexity this is not going to be an easy
addition, so don't hold your breath. Would love to review/merge patchs
adding this though…

In theory there's also the option of repurposing the hw watchdog for
this: i.e. instead of pinging it regularly, just set it to the maximum
shutdown timeout early, and then leave it unpinged for the rest of the
shutdown. However, given the limited configurable time range of most
hw watchdogs this doesn't appear realistic to me, and very hw
specific.

Lennart

-- 
Lennart Poettering, Red Hat