[systemd-devel] protecting sshd against forkbombs, excessive memory usage by other processes

[Tomasz Chmielewski]

On 2020-08-12 22:07, Mantas Mikulėnas wrote:
> On Wed, Aug 12, 2020 at 7:03 AM Tomasz Chmielewski <mangoo at wpkg.org>
> wrote:
> 
>> I've made a mistake and have executed a forkbomb-like task. Almost
>> immediately, the system became unresponsive, ssh session froze or
>> were
>> very slow to output even single characters; some ssh sessions timed
>> out
>> and were disconnected.
>> 
>> It was not possible to connect a new ssh session to interrupt the
>> runaway task - new connection attempt were simply timing out.
>> 
>> SSH is the only way to access the server. Eventually, after some 30
>> mins, the system "unfroze" - but - I wonder - can systemd help
>> sysadmins
>> getting out of such situations?
>> 
>> I realize it's a bit tricky, as there are two cases here:
>> 
>> 1) misbehaving program is a child process of sshd (i.e. user logged
>> in
>> and executed a forkbomb)
> 
> I don't think "child process of sshd" is the useful part, as logged-in
> user processes are actually moved to a separate cgroup for the session
> – so yes, they're sshd children, but they actually have resource
> limits fully separate from the main sshd daemon process.
> 
> Which means that with systemd, each user already has their own limit
> on the number of processes/tasks (the default in user-.slice.d is
> TasksMax=33% of...something, but it could be lowered to e.g. 10% or to
> 4096) without affecting the service itself.
> 
> So I'm sure that sshd.service and user-0.slice could be tweaked
> somehow to give root a higher priority at cgroup level, but that
> depends on what your system actually ran out of...

It ran out of memory.


Tomasz