[systemd-devel] journald reliability

[Zbigniew Jędrzejewski-Szmek]

On Thu, Oct 18, 2012 at 07:02:28PM +0200, Lennart Poettering wrote:
> On Thu, 18.10.12 16:10, Zbigniew Jędrzejewski-Szmek (zbyszek at in.waw.pl) wrote:
> > But systemd has/had this information and could be queried for it. So I
> > propose to add a cache of recently dead PIDs in systemd, that could be
> > queried by journald. I'm not sure about the details, but probably
> > keeping the information for a few seconds should be enough. This would
> > lead to more reliable logging for messages close to the end of the
> > program.
> 
> This wouldn't really work I fear, since systemd only really tracks the
> main and control PIds of a cgroup, and there might be much more that
> log.
> 
> I think the only real way to solve this cleanly is to extend the kernel
> to provide SCM_CREDENTIAL and SCM_SECURITY-like auxiliary messages that
> carries the information we need. More specifically I am hoping that we
> can get SCM_AUDIT (for the loginuid + sessionid of a process),
> SCM_EXECINFO (for exe, comm, cmdline), SCM_CGROUP (for cgroup
> membership). With that we should have all data we need in a safe and
> secure way.
So basically it all comes down to fixing the kernel...

> > I guess that some applications could simply block, but in case of e.g.
> > systemd itself it is not possible. journald should process the
> > messages faster. Should journald run reniced to higher priority? I
> > think that the problem is worse with journald than with syslog,
> > because journald tries to do more things: parse the message, query
> > cgroup information, forward the message to syslog, store it on disk.
> > syslog only does the last one, so necessarily must be more efficient.
> 
> I remember that maemo used to have issues with syslog blocking. They
> basically ran into a priority inversion problem, where RT threads ended
> up waiting for the non-RT syslog. I think large qlen are a pretty good
> solution for this.
OK, but let's say that we have 1000 processes generating
messages. journald would be overrun anyway, no matter what the queue
size actually is.

Zbyszek