[systemd-devel] watchdog and sd_notify between host and container systemd etc inits

[Lennart Poettering]

On Do, 26.09.19 07:24, Mikko.Rapeli at bmw.de (Mikko.Rapeli at bmw.de) wrote:

> Hi,
>
> I'd like to run systemd as init on host but run various containers and some of them
> with their own container side systemd init.
>
> Then I'd like to have sd_notify and watchdog available to check the health of the
> systemd init in the container. I trust the init in the container to check the health
> of all the services and processes running there.
>
> If systemd init in the container fails to respond to watchdog, then I'd like to
> restart only the container, not the whole host system.
>
> For the container systemd watchdog, I've proposed patch:
>
> https://github.com/systemd/systemd/pull/13643
>
> Comments to the PR mention that sd_notify support would be better, but AFAIK it uses
> the PID of processes and thus doesn't work with another systemd init as PID 0 in
> the container PID namespace.
>
> Thus we inveted a simple fifo between host init and container init where
> container writes MACHINE and HOSTNAME as watchdog ping. This works well with a
> custom watchdog manager on host and systemd init in an LXC container.
>
> These don't seem to fit very well to systemd, and we'd also like to know sd_notify type
> things like when is the container in running state, which systemd nspawn does
> provide, but I have use cases also for LXC containers...
>
> So, could you provide some ideas and/or feedback how this kind of functionality
> could/should be implemented with systemd?

I would normally assume that it's the job of the container manager to
watchdog/supervise its container. And the host systemd to supervise
the container manager in turn. i.e. it should be PID 1 on the host that gets
sd_notify() messages keep-alive messages from your container manager,
ensuring it's alive. And the container manager should get them from
PID in the container, and ensure it remains alive. And the PID 1 in
the container in the payload get the messages from the services below
it. i.e. instead of forwarding these messages across these boundaries
just have a clear 1:1 supervisor-client relationship.

Or in other words: try to convince the LXC maintainers to send out
sd_notify() watchdog messages from its own supervisor, and optionally
expect them from the payload PID 1. systemd-nspawn at least partly
works that way already.

Lennart

--
Lennart Poettering, Berlin