[systemd-devel] some services always being killed when stress tests running

[Han Pingtian]

On Wed, Apr 13, 2016 at 07:20:12PM +0000, Zizka, Jan (Nokia - CZ/Prague) wrote:
> > From: systemd-devel [mailto:systemd-devel-bounces at lists.freedesktop.org] On Behalf Of EXT Han Pingtian
> > Sent: Wednesday, April 06, 2016 4:33 AM
> > On Fri, Apr 01, 2016 at 09:13:54PM +0200, Lennart Poettering wrote:
> > > On Tue, 22.03.16 10:02, Han Pingtian (hanpt at linux.vnet.ibm.com) wrote:
> > >
> > > > But only after about 30 minutes, a lot of systemd services failed
> > > > and restarted like this:
> > > >
> > > > ... ...
> > > > [26885.910036] systemd[1]: systemd-journald.service: Failed with result
> > 'signal'.
> > > > [26885.910218] systemd[1]: systemd-udevd.service: Main process
> > > > exited, code=killed, status=9/KILL
> > >
> > > This indicates that something killed the processes in question with
> > > SIGKILL. Quite possibly this was the OOM killer, which was triggered
> > > by your stress test? Check the kernel logs if you see anything about
> > > that...
> > >
> > I have seen this problem on another system a while ago. But on all the
> > systems which this problem can be reproduced, there isn't any OOM killer
> > message can be found in kernel logs. How could we debug this problem?
> 
> You could use auditd to monitor the signals and then you will see which 
> process have sent the SIGKILL. There is also another method mentioned here:
> https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/Finding_the_source_of_signals_on_Linux_with_strace_auditd_or_Systemtap?lang=en
> 
> Jan

Thanks so much! I have find the reason by using systemtap. It's some
test cases from LTP kill the services.