[systemd-devel] some services always being killed when stress tests running

[Zizka, Jan (Nokia - CZ/Prague)]

> From: systemd-devel [mailto:systemd-devel-bounces at lists.freedesktop.org] On Behalf Of EXT Han Pingtian
> Sent: Wednesday, April 06, 2016 4:33 AM
> On Fri, Apr 01, 2016 at 09:13:54PM +0200, Lennart Poettering wrote:
> > On Tue, 22.03.16 10:02, Han Pingtian (hanpt at linux.vnet.ibm.com) wrote:
> >
> > > But only after about 30 minutes, a lot of systemd services failed
> > > and restarted like this:
> > >
> > > ... ...
> > > [26885.910036] systemd[1]: systemd-journald.service: Failed with result
> 'signal'.
> > > [26885.910218] systemd[1]: systemd-udevd.service: Main process
> > > exited, code=killed, status=9/KILL
> >
> > This indicates that something killed the processes in question with
> > SIGKILL. Quite possibly this was the OOM killer, which was triggered
> > by your stress test? Check the kernel logs if you see anything about
> > that...
> >
> I have seen this problem on another system a while ago. But on all the
> systems which this problem can be reproduced, there isn't any OOM killer
> message can be found in kernel logs. How could we debug this problem?

You could use auditd to monitor the signals and then you will see which 
process have sent the SIGKILL. There is also another method mentioned here:
https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/Finding_the_source_of_signals_on_Linux_with_strace_auditd_or_Systemtap?lang=en

Jan