[systemd-devel] protecting sshd against forkbombs, excessive memory usage by other processes

[Mantas Mikulėnas]

On Wed, Aug 12, 2020 at 7:03 AM Tomasz Chmielewski <mangoo at wpkg.org> wrote:

> I've made a mistake and have executed a forkbomb-like task. Almost
> immediately, the system became unresponsive, ssh session froze or were
> very slow to output even single characters; some ssh sessions timed out
> and were disconnected.
>
> It was not possible to connect a new ssh session to interrupt the
> runaway task - new connection attempt were simply timing out.
>
> SSH is the only way to access the server. Eventually, after some 30
> mins, the system "unfroze" - but - I wonder - can systemd help sysadmins
> getting out of such situations?
>
> I realize it's a bit tricky, as there are two cases here:
>
> 1) misbehaving program is a child process of sshd (i.e. user logged in
> and executed a forkbomb)
>

I don't think "child process of sshd" is the useful part, as logged-in user
processes are actually moved to a separate cgroup for the session – so yes,
they're sshd children, but they actually have resource limits fully
separate from the main sshd daemon process.

Which means that with systemd, each user already has their own limit on the
number of processes/tasks (the default in user-.slice.d is TasksMax=33%
of...something, but it could be lowered to e.g. 10% or to 4096) without
affecting the service itself.

So I'm sure that sshd.service and user-0.slice could be tweaked somehow to
give root a higher priority at cgroup level, but that depends on what your
system actually ran out of...

-- 
Mantas Mikulėnas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20200812/e24371f0/attachment.htm>