[systemd-devel] journald reliability

[Lennart Poettering]

On Fri, 19.10.12 00:26, Zbigniew Jędrzejewski-Szmek (zbyszek at in.waw.pl) wrote:

> > I think the only real way to solve this cleanly is to extend the kernel
> > to provide SCM_CREDENTIAL and SCM_SECURITY-like auxiliary messages that
> > carries the information we need. More specifically I am hoping that we
> > can get SCM_AUDIT (for the loginuid + sessionid of a process),
> > SCM_EXECINFO (for exe, comm, cmdline), SCM_CGROUP (for cgroup
> > membership). With that we should have all data we need in a safe and
> > secure way.
>
> So basically it all comes down to fixing the kernel...

Yeah, unfortunately.

> > I remember that maemo used to have issues with syslog blocking. They
> > basically ran into a priority inversion problem, where RT threads ended
> > up waiting for the non-RT syslog. I think large qlen are a pretty good
> > solution for this.
>
> OK, but let's say that we have 1000 processes generating
> messages. journald would be overrun anyway, no matter what the queue
> size actually is.

That is a valid concern. But at one point you *must* make a cut, and say
"OK, this is now getting waaaaay too much, now it is better to drop
rather than collecting more and more and more and DoSing the system for
good". Which is what we do, though admittedly much too early, but where
adding a socktopt for the queue size would help.

That said, we probably could improve this worst case scenario a bit. For
example, for the syslog forwarding we recently added a scheme where we'd
count how much we dropped and we would log this counter as soon as the
queue gets unclogged. We did this on request of the HA guys who
basically said "it's OK if you drop, but you need to tell us about
that". So maybe we shoud apply this to log.c and journal-send.c as well,
and instead of queing locally we should just count and flush a messages
about that when things get unclogged?

(added this to the TODO list now)

Lennart

-- 
Lennart Poettering - Red Hat, Inc.