[systemd-devel] User question: how to manage an application split up into multiple binaries The Right Way.

[Etienne Doms]

On Wed, 5 Jun 2019 at 10:56, Reindl Harald <h.reindl at thelounge.net> wrote:
>
>
>
> Am 05.06.19 um 10:34 schrieb Etienne Doms:
> > Hi all,
> >
> > We are developing an application on an embedded system which, for some
> > reasons, must be split up into multiple processes. All these processes
> > are linked between each other, they communicate using named pipes, and
> > all of them should be considered as a single entity materializing the
> > application. If one process goes down, the whole application goes
> > down, this is not an issue.
> >
> > We have the following requirements:
> >
> > - The whole system shall be monitored by a hardware watchdog.
> > - Each process shall be monitored by a software watchdog.
> > - If one process aborts, all processes should be restarted.
> > - If one process aborts more than x times, the whole system shall be rebooted.
>
> in theory
>
> * each process a own service type=notify
>   https://www.freedesktop.org/software/systemd/man/sd_notify.html
>
> * proper ordering
>
> * Restart=on-failure
>

Yep, this what I have for now.

However, I have no ordering issue since all processes communicate
using named pipes. When spawned, they first open the readers (non
blocking) and then the writers (blocking), so everybody magically
synchronize on this sequence.

> --------------------
>
> but "if one process all processes" makes it a little hard, maybe it
> would be better to have a master process like httpd or postfix which is
> type=notify and forking/monitoring the other ones and keep it as one
> service and if that only monitoring process fails reboot while it only
> controls and restarts the workers

This is indeed the second solution if I cannot fulfill all our
requirements using only pure systemd mechanisms, but we'd like to
avoid that, I won't go into the details but we have heavy and
complicated procedures and each new line of code is really expensive
:)

My goal is to deport as much "system requirements" to
systemd/libsystemd features as possible.

Actually, here's what I have, for now:

- app.target
- app-process1.service
- app-process2.service
- app-process3.service

And here's what I want to achieve:

- I want the whole application to be protected by a hardware watchdog.
This works using RuntimeWatchdogSec in system.conf.
- I want to be able start and stop all services with a single command.
This works using Requires directives in app.target and PartOf
directives in each app-process*.service.
- I want all app-process*.service to be monitored by a software
watchdog. This works using WatchdogSec and libsystemd API.
- I want all app-process*.service to be restarted when failing, or a
complete reboot if a limit is reached. This works using Restart and
FailureAction directives in each app-process*.service
- I want all app-process*.service to be restarted if one of them
fails. For now I rely on the broken pipes to make everybody abort and
be automatically restarted (or hard reboot if limit reached), but I
was looking for a better way.

At first, I expected to be able to see the target going to "failed"
state if one of its required services goes down, and put a Restart
directive on the target itself to restart all dependencies, but
Restart is a service property, and can't be set on a target. It makes
sense since Restart depends on the exit code of a process, which is
not applicable to a target, but I'm still looking for a smart way to
tell systemd "restart a, b and c if either a, b or c fails".

Thank you for your quick reply!

>
> https://www.freedesktop.org/software/systemd/man/sd_notify.html
> https://www.freedesktop.org/software/systemd/man/systemd.service.html
> https://www.freedesktop.org/software/systemd/man/systemd.unit.html
>
> these options should give you some start
>
> Restart=
> FailureAction=
> StartLimitAction=
> StartLimitIntervalSec=
> StartLimitBurst=
>
> _______________________________________________
> systemd-devel mailing list
> systemd-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel