[systemd-devel] systemd debug out of memory

[Lennart Poettering]

On Tue, 28.02.17 13:26, Pascal kolijn (p.kolijn at vu.nl) wrote:

> Hi List,
> 
> I've subscribed to this list to ask for help in debugging a problem we
> seem to have with the socket activated telnetd on a rhel7 system.
> 
> A default install of telnetd collects data from some small boxes
> deployed in the field. It works for a long time and then suddenly:
> 
> Feb 26 17:46:53 bibr systemd: Created slice user-6006.slice.
> Feb 26 17:46:53 bibr systemd: Starting user-6006.slice.
> Feb 26 17:46:53 bibr systemd: Started Session 2223341 of user <USER>.
> Feb 26 17:46:53 bibr systemd-logind: New session 2223341 of user <USER>.
> Feb 26 17:46:53 bibr systemd: Starting Session 2223341 of user <USER>.
> Feb 26 17:46:53 bibr systemd: Started Telnet Server (<IP>:28830).
> Feb 26 17:46:53 pbibr001 systemd: Starting Telnet Server (<IP>:28830)...
> Feb 26 17:46:57 bibr systemd: Failed to fork: Cannot allocate memory

Hmm, Linux fork() returns ENOMEM if the maximum number of tasks on the
system is hit (yes this is a bit misleading, but that's how it is).
That max number of tasks is limited for example by the max number of
assignable pids as configured in /proc/sys/kernel/pid_max? Maybe you
hit that limit? Maybe something is leaking pids on your system? not
reaping zombies properly?

> Feb 26 17:46:57 bibr systemd: Assertion 'pid >= 1' failed at
> src/core/unit.c:1996, function unit_watch_pid(). Aborting.
> Feb 26 17:46:57 bibr001 systemd: Caught <ABRT>, cannot fork for core
> dump: Cannot allocate memory
> Feb 26 17:46:57 bibr systemd: Freezing execution.

So this is definitely a bug. If the limit is hit, we hould certainly
not hit an assert. I tried to figure out how this could ever happen,
but afaics this should not be possible on current git at least. Any
chance you can try to reproduce this isue with something more recent
than a rhel7 box?

Either way it appears that there's both a bug on your setup and in
systemd: something leaks processes (which is bug #1, in your setup)
and then systemd doesn't deal properly with that (which is bug #2, in
systemd upstream)...

Lennart

-- 
Lennart Poettering, Red Hat