[systemd-devel] System stability when journald locks up

[Marti Raudsepp]

On Wed, May 30, 2012 at 10:59 PM, Lennart Poettering
<lennart at poettering.net> wrote:
> Many people have configured their classic syslog daemon to output logs
> on /dev/tty12. If you press C-s there (or accidentally hit Scroll Lock)
> you end up freezing syslog too and thus freezing the entire machine
> sooner or later. It's kind of a known problem.

Oh, for some reason I was under the impression that this problem was
caused by journald. But you're right, all my systems are already at
the mercy of syslogd. Sorry, I didn't mean to be unfair. :)

> Adding the timeout change there (which would actually be dead-easy,
> simply by using SO_SNDTIMEO) would not really fix the problem too well
> though: given the amount of messages that are generated the system
> might not be locked up entirely but still very slow.

True. Or O_NONBLOCK, but then we'd start dropping messages as soon as
the kernel packet queue fills up.

----

Well, one potential solution would be to spawn a thread whose only job
is to pop messages from /dev/log and copy them to a larger user-space
buffer, shared with journald. If this memory buffer fills up, we know
for a fact that an application is sustainably generating more messages
than we can write out, so dropping them might be a good idea to
prevent journald from becoming the bottleneck. Additionally, journald
could emit a notice about the dropped messages.

On the one hand, this could in theory be used by an attacker to
conceal log messages (s)he doesn't want the administrator to see.
On the other hand, it would alleviate a local DoS hole that can
currently be used to slow down the machine.

Regards,
Marti