[systemd-devel] Thoughts about storing unit/job statistics

[Philip Withnall]

On Thu, 2019-11-28 at 09:32 +0100, Lennart Poettering wrote:
> On Mi, 27.11.19 14:26, Philip Withnall (philip at tecnocode.co.uk)
> wrote:
> 
> > > > If I were to implement this as a separate daemon, it would need
> > > > to
> > > > be
> > > > active all the time, listening to
> > > > UnitNew/UnitRemoved/JobNew/JobRemoved
> > > > signals from systemd. That seems like a waste of a process.
> > > > Let’s
> > > > call
> > > > this problem 0.
> > > 
> > > This data is already collected and written to the journal anyway
> > > if
> > > you turn on the various XYZAccounting= properties for your
> > > services. Then use the invocation ID of the service and the
> > > MESSAGE_ID=ae8f7b866b0347b9af31fe1c80b127c0 match in journalctl
> > > to
> > > find these.
> > 
> > We’re interested in the wall clock time that a unit/scope was
> > active,
> > not the CPU time, so I suspect we’d have to add another message
> > along
> > the same lines.
> 
> Just add another structured field to the existing message. The
> message
> already contains IO/CPU/IP/… stats, hence adding more time stats
> definitely makes sense.

OK, I’ll put a merge request together for that sometime soon, since
that looks easy, useful and self-contained.

> > > A long-standing TODO item in systemd was to have some form of
> > > metrics
> > > collector, that may be turned on and that writes a time-keyed
> > > ring
> > > buffer  of metrics collected per service to disk, from the data
> > > collected via cgroup attributes, bpf and so on. But so far noone
> > > has
> > > found the time to do it. It probably should be decoupled from PID
> > > 1
> > > in
> > > some form, so that PID 1 only pings it whenever a new cgroup
> > > shall be
> > > watched but the collecting/writing of the data points is done
> > > entirely
> > > separate from it.
> > 
> > Would the idea with that be that it uses the journal, or not? Is
> > there
> > a task in GitHub for it?
> 
> In my current thinking it would be similar to journald in many way,
> but not be journald, since the data is differently structured
> (i.e. not keyed by arbitrary fields but keyed by time, just time-
> based
> ring buffers). The idea would be to mantain ring buffers in /run/ and
> /var/ similar to how journald does it, and have "systemd-metricsd"
> pull at its own pace metrics from the various cgroups/bpf tables/…
> and
> write them to these buffers. Apps could then mmap them and pull the
> data out either instantly (if they are located in /run/) or after
> substantial latency (if they are located in /var/) depending on the
> usecase.
> 
> Ideally we wouldn't even come up with our own file format for these
> ring buffers, and just use what is already established, but afaiu
> there's no established standard for time series ring buffer files so
> far, hence I figure we need to come up with our own. I mean, after
> all
> the intention is not to process this data ourselves but have other
> tools do that.
> 
> There's #10229.

Thanks for the info and reference. I’ll continue to ponder about which
approach I’ll go with, based on the time/effort required to solve the
immediate problem for the desktop parental controls feature (which is
quite a small subset of what a full ring buffer store of unit
statistics would provide).

Ta,
Philip