[systemd-devel] Looping too fast. Throttling execution a little

[Thierry Parmentelat]

Hi there


---
I am running a fedora21 box that acts as a libvirt host for lxc containers guests 
This is based on vanilla fedora (maybe a little bit behind by a few weeks) and a locally rebuild libvirt based on 1.2.11


root at bigjohn /var/log # uname -a
Linux bigjohn.pl.sophia.inria.fr 3.18.3-201.fc21.x86_64 #1 SMP Mon Jan 19 15:59:31 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

root at bigjohn /var/log # rpm -q libvirt
libvirt-1.2.11-1.x86_64


guests typically are based on fedora20, fedora14, and debian jessie

—
The symptom I have is, after some time (typically a week) guests become real slow to answer; trying to ssh into a guest takes in the 5 to 10 seconds. 

The most visible of these guests runs feodra20; when this problem occurs I can generally get the guest to answer again with its expected pace by just restarting the guest 

I have been seeing this for quite some time - something like 6-9 months, starting with the host running fedora20/libvirt-1.2.5
I was hoping this would go away with fedora21, but that is not the case


—
Digging a little more today, I found the following on the host side

As there was a message on this list in december named
[systemd-devel] Cannot use systemctl after heavy swapping
I figured I would report what I can see on my host





root at bigjohn /vservers # journalctl -b -f
-- Logs begin at Fri 2014-09-05 04:28:57 CEST. --
Feb 18 10:38:25 bigjohn.pl.sophia.inria.fr systemd[1]: Looping too fast. Throttling execution a little.
Feb 18 10:38:26 bigjohn.pl.sophia.inria.fr systemd[1]: Looping too fast. Throttling execution a little.
Feb 18 10:38:28 bigjohn.pl.sophia.inria.fr systemd[1]: Looping too fast. Throttling execution a little.
Feb 18 10:38:30 bigjohn.pl.sophia.inria.fr systemd[1]: Looping too fast. Throttling execution a little.
Feb 18 10:38:31 bigjohn.pl.sophia.inria.fr systemd[1]: Looping too fast. Throttling execution a little.
Feb 18 10:38:33 bigjohn.pl.sophia.inria.fr systemd[1]: Looping too fast. Throttling execution a little.



root at bigjohn /vservers # systemctl list-unit-files
<hangs for ages>
^C 



# strace -p 1 
<…>
recvmsg(42, 0x7fff3ea64d00, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
timerfd_settime(3, TFD_TIMER_ABSTIME, {it_interval={0, 0}, it_value={0, 1}}, NULL) = 0
epoll_wait(4, {{EPOLLIN, {u32=3, u64=3}}}, 36, 0) = 1
clock_gettime(CLOCK_BOOTTIME, {1959524, 957887776}) = 0
read(3, "\1\0\0\0\0\0\0\0", 8)          = 8
recvmsg(42, 0x7fff3ea64d00, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
timerfd_settime(3, TFD_TIMER_ABSTIME, {it_interval={0, 0}, it_value={0, 1}}, NULL) = 0
epoll_wait(4, {{EPOLLIN, {u32=3, u64=3}}}, 36, 0) = 1
clock_gettime(CLOCK_BOOTTIME, {1959524, 958282696}) = 0
read(3, "\1\0\0\0\0\0\0\0", 8)          = 8
recvmsg(42, 0x7fff3ea64d00, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
timerfd_settime(3, TFD_TIMER_ABSTIME, {it_interval={0, 0}, it_value={0, 1}}, NULL) = 0
epoll_wait(4, {{EPOLLIN, {u32=3, u64=3}}}, 36, 0) = 1
clock_gettime(CLOCK_BOOTTIME, {1959524, 958617432}) = 0
^CProcess 1 detached




——
I’d very much like to help in providing more data when this problem occurs again (although I will reboot the host right now I am afraid)

Just let me know what commands I should do if I see this again

Thanks — Thierry