[systemd-devel] Complex supervision structures/delegating watchdog?

[Lennart Poettering]

On Mon, 16.02.15 21:36, Holger Hans Peter Freyther (holger at freyther.de) wrote:

Sorry for the late reply, still catching up with piles of email queued
up.

> On Mon, Feb 16, 2015 at 11:21:53AM +0100, Lennart Poettering wrote:
> > 
> > Is your pppd daemon itself also a systemd service?
> > 
> > What precisely does "monitor" consist of for this case?
> 
> Yes, the pppd is invoked by systemd as a service. What I am
> doing right now is:
> 
> link.service:
> ExecStart=pppd call uplink ... nodetach
> 
> /etc/ppp/ip-up.d/NN-linkmon:
> linkmon -i $PPP_IFACE -d 8.8.8.8 -p `cat /run/${PPP_IFACE}.pid` &
> 
> 
> The monitoring right now involves simple ICMP request and
> depending on the outcome I change the metric of one of the
> default routes. So in case some amount of packet loss is
> reached the linkmon will SIGKILL the pid provided.
> 
> > We have watchdog support already, with sd_notify(0, "WATCHDOG=1"), and
> > WatchdogSec=. But that requires you to run your pppd as a service of
> > its own, to be useful.
> 
> I thought the sd_notify is only possible by the "main"
> application that got started? 

Well, that's the way this was intended originally, but you actually
use it differently, by using NotifyAccess=all (instead of
NotifyAccess=main).

That said given your usecase I don#t really see how the watchdog
systemd implements would be helpful. After all the key words here
really are "app specific monitoring" rather than just "ping watchdog".

So far whenever the question of monitoring came up I just said: we
provide the apis, you implement the monitoring, without going into
more detail. I figure we should put together some wiki page that
explains exactly how we think a monitoring tool would make use of
systemd's functionality, and actually figure out if we really have
everything in place to make it work nicely, and fill the gaps where we
don't.

> E.g. in the above case the linkmon would be a child of pppd. My
> application wouldn't run until pppd has setup the link. This means I
> would need to configure a high enough timeout to cope with a
> potential bigger set-up time.

Ideally, pppd would simply report back when it finished setting up the
connection. Then, the monitoring daemon could use After= on the ppd
service, and would be started only after the connection up. And it
should use BindsTo= to glue itself to the pppd instances.

I figure making pppd report back when it completed setting up the
connection is easy, by making use of the normal "detach" mode with
Type=forking and a PID file. Even better of course would be to patch
pppd to use sd_notify("READY=1") so that this ugliness with double
forking would be unnecessary

Hope this makes some sense,

Lennart

-- 
Lennart Poettering, Red Hat