[systemd-devel] Hardware watchdog support, slug speed.

[Sébastien Luttringer]

Hello,

I thought that systemd and the watchdog driver sbc_fitpc2_wdt was not
friends. I use the both on an archlinux i686 setup to do some network
basics.
Everything works pretty well with watchdogd, I doesn't use the
advanced features and systemd would do the job perfectly. Hum. But
not.

The issue, is not a matter of support, because systemd (and the whole
things communicating with it) become very slow when I enable the
watchdog.

Some metrics:

systemctl call, without watchdog enabled : 0s
# time systemctl --no-page
systemctl --no-pager  0,00s user 0,01s system 27% cpu 0,049 total

systemctl call, with watchdog enabled.: 4s
# time systemctl --no-page
systemctl --no-pager  0,01s user 0,01s system 0% cpu 4,359 total

whole system boot-up without watchdog : 5s
# systemd-analyze
Startup finished in 1987ms (kernel) + 2884ms (userspace) = 4871ms

whole system boot-up with watchdog : 2min57s (aye!!!)
# systemd-analyze
Startup finished in 2154ms (kernel) + 175579ms (userspace) = 177733ms

Even the call to systemd-analyse is longer. Firstly I guess that the
driver is slowing the whole system, but only systemd is slow and when
using watchdogd, systemd is fast.

I use a simple configuration

# grep Watch /etc/systemd/system.conf
RuntimeWatchdogSec=1min
ShutdownWatchdogSec=2min

After reexecuting the daemon, I got the following info in the journal.

# systemctl daemon-reexec
# LC_ALL=C journalctl -f
mars 11 20:50:15 slice systemd[1]: systemd 198 running in system mode.
(+PAM -LIBWRAP -AUDIT -SELINUX -IMA -S...L +XZ)
mars 11 20:50:15 slice systemd[1]: Hardware watchdog 'SBC-FITPC2
Watchdog', version 0
mars 11 20:50:16 slice systemd[1]: Set hardware watchdog to 1min.
mars 11 20:50:16 slice systemd[1]: Failed to enable hardware watchdog:
Inappropriate ioctl for device

No load average.
# LC_ALL=C uptime
 21:11:29 up 1 day, 23:12,  3 users,  load average: 0.13, 0.18, 0.22

No  tremendous read/write.
# dstat
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
  1   1  98   0   0   0| 650B 3219B|   0     0 |   0     0 | 676   421
  1   1  98   0   0   0|   0     0 | 162k  315k|   0     0 | 477   317

No process running at 100%. But systemd act as a slug.

Stracing pid 1, there is (even if the message in jounal is failed to
enable) an ioctl evey 30s has expected.

# strace -t -e ioctl -p 1
Process 1 attached
22:09:28 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0
22:09:59 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0
22:10:30 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0
22:11:00 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0

But everytime I call a systemd command, the ioctl is called. Here is
the output of the same strace, with a call on systemctl --no-pager.

# strace -t -e ioctl -p 1
Process 1 attached
22:12:41 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0
22:12:44 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0
22:12:44 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0
22:12:45 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0
22:12:46 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0
22:12:46 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0
22:12:47 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0
22:12:48 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0
22:12:48 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0
22:12:49 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0
22:12:49 ioctl(38, WDIOC_KEEPALIVE, 0)  = 0

I guess issue come from the manager_loop (src/core/manager.c) which
call the watchog_ping each loop iteration.
Even when there is successive and positive return from manager_dispatch_*.

Seeing that's I figure that all my system with watchdog enabled are
"slugged" by watchdog activation. And it's true.
All my computers call ioctl more than 10th by second. But with
powerful intel modern computer I doesn't see it since weeks.

So, I thnk this code should be reworked and only do the ioctl at
interval defined in RuntimeWatchdogSec, firstly because we use too
many cycle (global speed/power) and
secondly, because there is no way to trigger a system reset by the
wathdog in these condition.
I'm also interogative about calling ioctl (and trying to reopen
/dev/watchdog) even when the initial setup of the watchdog have
failed.

I could provide some patches to fix this if you are interested.

Cheers,

-- 
Sébastien "Seblu" Luttringer
https://www.seblu.net
GPG: 0x2072D77A