[systemd-devel] systemd debug out of memory

[Lennart Poettering]

On Sun, 05.03.17 15:59, Pascal Kolijn (p.kolijn at vu.nl) wrote:

> Peace,
> 
> On 28/02/2017 16:00, Lennart Poettering wrote:
> > On Tue, 28.02.17 13:26, Pascal kolijn (p.kolijn at vu.nl) wrote:
> > 
> >> Hi List,
> >>
> >> I've subscribed to this list to ask for help in debugging a problem we
> >> seem to have with the socket activated telnetd on a rhel7 system.
> >>
> >> A default install of telnetd collects data from some small boxes
> >> deployed in the field. It works for a long time and then suddenly:
> >>
> >> Feb 26 17:46:53 bibr systemd: Created slice user-6006.slice.
> >> Feb 26 17:46:53 bibr systemd: Starting user-6006.slice.
> >> Feb 26 17:46:53 bibr systemd: Started Session 2223341 of user <USER>.
> >> Feb 26 17:46:53 bibr systemd-logind: New session 2223341 of user <USER>.
> >> Feb 26 17:46:53 bibr systemd: Starting Session 2223341 of user <USER>.
> >> Feb 26 17:46:53 bibr systemd: Started Telnet Server (<IP>:28830).
> >> Feb 26 17:46:53 pbibr001 systemd: Starting Telnet Server (<IP>:28830)...
> >> Feb 26 17:46:57 bibr systemd: Failed to fork: Cannot allocate memory
> > 
> > Hmm, Linux fork() returns ENOMEM if the maximum number of tasks on the
> > system is hit (yes this is a bit misleading, but that's how it is).
> > That max number of tasks is limited for example by the max number of
> > assignable pids as configured in /proc/sys/kernel/pid_max? Maybe you
> > hit that limit? Maybe something is leaking pids on your system? not
> > reaping zombies properly?
> 
> As far as I can determine running out of pids is not the issue, as I can
> see pids being reused in a day, which will not say that some may still
> go missing over time, but how do I determine if that is the case...?

Well, the "Failed to fork" log message is generated when "fork()"
returns ENOMEM, i.e. this is generated by a syscall, i.e. it's an
allocation of some object in the kernel that fails, not of userspace...

> What I do see is that the rss of the systemd process is slowly growing
> over time in the production environment. I've not been able (yet) to
> reproduce the situation in a test environment, which is a pity. I think
> I can simulate the telnet connects more accurately after I speak with
> the developer of the said boxes, and see if I can create a reproducible
> situation.

Well, if it grows without bounds then this would be a leak. But if it
grows only a bit during runtime that's not a problem: in-process
memory management generally doesn't return all memory back to the
kernel immediately, but keeps an allocation cache. In fact, our
hashtable implementation in systemd (which is heavily used) implements
an extra allocation cache on top. Memory in that cache is not actually
leaked, even though it is not returned to the OS: it is simply kept
around for next time we need a similar object.

Lennart

-- 
Lennart Poettering, Red Hat