[systemd-devel] protecting sshd against forkbombs, excessive memory usage by other processes

[Benjamin Berg]

Hi,

I would suggest trying the following:

 * Set a MemoryLow allocation
 * Enable the CPU cgroup controller

For the first, it'll make sense to set MemoryLow= on system.slice and
also setting DefaultMemoryLow= or MemoryLow= on sshd.service. Otherwise
things might be somewhat unexpected for now, see
  https://github.com/systemd/systemd/pull/16559
I guess one could also do something similar for user-0.slice.

The second part ensures CPU is allocated to users fairly, meaning that
the user-X.slice's are competing against each other, rather than the
individual processes. This will effectively give the root login and SSH
service a higher CPU priority in relation to the fork bomb. You can do
this by setting CPUWeight=100 on user-.slice. It'll also result in
system.slice and user.slice competing for CPU at eye level.

Benjamin

On Wed, 2020-08-12 at 12:57 +0900, Tomasz Chmielewski wrote:
> I've made a mistake and have executed a forkbomb-like task. Almost 
> immediately, the system became unresponsive, ssh session froze or
> were 
> very slow to output even single characters; some ssh sessions timed
> out 
> and were disconnected.
> 
> It was not possible to connect a new ssh session to interrupt the 
> runaway task - new connection attempt were simply timing out.
> 
> SSH is the only way to access the server. Eventually, after some 30 
> mins, the system "unfroze" - but - I wonder - can systemd help
> sysadmins 
> getting out of such situations?
> 
> I realize it's a bit tricky, as there are two cases here:
> 
> 1) misbehaving program is a child process of sshd (i.e. user logged
> in 
> and executed a forkbomb)
> 
> 2) misbehaving program is not a child process of sshd (i.e. some
> system 
> service is using a lot of resources)
> 
> 
> Given that - how can we tune systemd so that system admin is almost 
> always able to log in via a new SSH connection, in both cases
> outlined 
> above? My usage case assumes user error rather than a malicious
> system 
> resource usage.
> 
> 
> 
> Tomasz Chmielewski
> _______________________________________________
> systemd-devel mailing list
> systemd-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20200814/af5c0f12/attachment.sig>