[systemd-devel] User question: how to manage an application split up into multiple binaries The Right Way.

[Etienne Doms]

Hi all,

We are developing an application on an embedded system which, for some
reasons, must be split up into multiple processes. All these processes
are linked between each other, they communicate using named pipes, and
all of them should be considered as a single entity materializing the
application. If one process goes down, the whole application goes
down, this is not an issue.

We have the following requirements:

- The whole system shall be monitored by a hardware watchdog.
- Each process shall be monitored by a software watchdog.
- If one process aborts, all processes should be restarted.
- If one process aborts more than x times, the whole system shall be rebooted.

Our platform is Linux with systemd as init manager, we are instructed
to avoid as many third-party libraries as possible and write as little
code as possible (we need 100% code coverage and expensive stuff like
that...), so I decided to use libsystemd to implement each process,
and feel like I don't need to specify/design/implement/test a master
process.

I have created a service file for each process, and use the built-in
systemd watchdog features (global RuntimeWatchdogSec and per service
WatchdogSec) to implement the first two requirements, it works great.

Now, I'm brainstorming about the two remaining ones. I somewhat need
to group all these processes into a single "abstract entity". I ended
up with a target file listing all services as Required, referenced by
all service files using PartOf. That way I can manually start and stop
the whole set of processes using the target. However, I read that
PartOf is one way dependency, so if one service aborts and restart
(using Restart=on-failure), the restart is not propagated to all
services required by the target listed as PartOf. Is there a systemd
way to propagate the restart?

I'm also reading about slices, which might also meet my requirements,
with the additional possibility to isolate all processes into a
dedicated CGroup, which looks awesome. Does it make sense? Is it
better than using a target?

What is your feeling about this?

Have I missed something obvious and/or am I over-engineering?

Thank you for your support.

Best regards,
Etienne Doms