[systemd-devel] Error : cannot fork for core dump: Cannot allocate memory

[Lennart Poettering]

On Sun, 31.07.16 08:42, P.R.Dinesh (pr.dinesh at gmail.com) wrote:

> My system was running for 2 days continuously and suddenly I started
> getting the following error
> 
> Broadcast message from systemd-journald at Acce-01--5712 (Thu 2016-07-28
> > 19:02:14 UTC):
> >  systemd[1]: Caught <ABRT>, cannot fork for core dump: Cannot allocate
> > memory
> >  Message from syslogd at Acce-01--5712 at Jul 28 19:02:14 ...
> >  systemd[1]: Caught <ABRT>, cannot fork for core dump: Cannot allocate
> > memory
> >  Message from syslogd at Acce-01--5712 at Jul 28 19:02:14 ...
> >  systemd[1]: Freezing execution.
> >  Broadcast message from systemd-journald at Acce-01--5712 (Thu 2016-07-28
> > 19:02:14 UTC):
> >  systemd[1]: Freezing execution.
> 
> 
> My guess is that the system ran out of memory and systemd is not able to
> fork new process (systemd-coredump) to perform the coredump operation.
> 
> Is my understanding correct?

I don't think so, no.

So, when PID 1 gets a SIGABRT or SIGSEGV or something like
that it will fork() once, freeze in the parent, and then raise the
signal again in the child. It does that, since the kernel halts the
system abruptly if PID 1 terminates. To avoid such an abrupt halt,
we'll leave PID 1 hang around, even on crash.

In your case it appears that the fork() call after a SIGABRT
failed. It's not clear to my why that could happen.

Is this some weird architecture? Or plain x86/x86-64?

> Also how should we handle this situation?  what is the expected behavior of
> systemd-coredump when the system ran out of resource.

Well, the right approach is to figure out why SIGABRT was caught by
PID1. This usually means some form of assert() was hit in PID1, and
the question is why.

Note that systemd code calls assert() a lot, but only for checking for
programmer errors, not for runtime errors. This hence suggests
something is severely wrong on your system.

Lennart

-- 
Lennart Poettering, Red Hat