[systemd-devel] Monitoring unit and overall state

[Lennart Poettering]

On Fr, 05.06.20 14:42, Tomáš Pecka (peckato1 at fit.cvut.cz) wrote:

> Hello,
>
> we are currently working on a project where we'd like to let users know
> the health status of the system's services. The health status is
> supposed to be considered as "is every unit in systemd ok?".
> For example, when a daemon crashes we want to switch the red light on
> and keep it until the service is started again. After the service is
> active again (and nothing else failed meanwhile), switch the green light on.
>
> At first, we thought that it will be sufficient to monitor manager's
> SystemState (or NFailedUnits) property on dbus. It sets up state to
> 'degraded' (and NFailedUnits=1) when the daemon crashes. However, the
> unit can be set to auto-restart, and when it does, the SystemState
> immediately restores back to 'running' (because the unit is in
> 'activating auto-restart' state and not failed anymore). So I don't
> think I can use it here.
>
> I can probably think of some ways how to achieve it. Perhaps monitoring
> ActiveState/SubState properties of all units and look for the unit
> states that we consider bad (probably 'failed' and 'activating
> auto-restart'?) and keep track of units that are (not) OK. I am not sure
> whether this is a good idea though. I think there might be better ways
> which I do not currently see.
>
> So -- is there a better way how get such information from systemd? We do
> not care about the output when the manager is starting/stopping.

A failing unit creates a journal log message of
MESSAGE_ID=d9b373ed55a64feb8242e02dbe79a49c you can thus do something
like thist:

journalctl MESSAGE_ID=d9b373ed55a64feb8242e02dbe79a49c -b -f

(or something equivalent by linking against libsystemd.)

It will report all failed services to you. You can even add "-o
verbose" (or -o json) to get additional structure info from it.

Lennart

--
Lennart Poettering, Berlin