[systemd-devel] Looping too fast. Throttling execution a little

[Florian Bruhin]

Hi,

I've just seen a bug happen similiar to [1] and [2]. My machine was
out of memory (and qemu launched via libvirt was killed by the OOM
killer), and since then I got "Cannot use systemctl after heavy
swapping" a lot in my journal.

strace'ing PID 1 showed this repeating pattern:

    clock_gettime(CLOCK_BOOTTIME, {1854741, 780810077}) = 0 
    read(14, "\1\0\0\0\0\0\0\0", 8)         = 8
    recvmsg(71, 0x7ffcc8bf92e0, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
    timerfd_settime(14, TFD_TIMER_ABSTIME, {it_interval={0, 0}, it_value={1854768, 287736000}}, NULL) = 0
    epoll_wait(4, {{EPOLLOUT, {u32=1112296432, u64=139729283337200}}, {EPOLLOUT, {u32=1113026480, u64=139729284067248}}}, 34, 0) = 2
    clock_gettime(CLOCK_BOOTTIME, {1854741, 787588006}) = 0
    recvmsg(42, 0x7ffcc8bf92e0, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
    epoll_wait(4, {{EPOLLOUT, {u32=1112296432, u64=139729283337200}}, {EPOLLOUT, {u32=1113026480, u64=139729284067248}}}, 34, 0) = 2
    clock_gettime(CLOCK_BOOTTIME, {1854741, 787720058}) = 0 
    recvmsg(40, 0x7ffcc8bf92e0, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
    timerfd_settime(14, TFD_TIMER_ABSTIME, {it_interval={0, 0}, it_value={0, 1}}, NULL) = 0
    epoll_wait(4, {{EPOLLOUT, {u32=1112296432, u64=139729283337200}}, {EPOLLOUT, {u32=1113026480, u64=139729284067248}}, {EPOLLIN, {u32=3, u64=3}}}, 34, 0) = 3
    clock_gettime(CLOCK_BOOTTIME, {1854741, 787846139}) = 0
    read(14, "\1\0\0\0\0\0\0\0", 8)         = 8
    recvmsg(42, 0x7ffcc8bf92e0, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
    timerfd_settime(14, TFD_TIMER_ABSTIME, {it_interval={0, 0}, it_value={1789574, 287736000}}, NULL) = 0
    epoll_wait(4, {{EPOLLOUT, {u32=1112296432, u64=139729283337200}}, {EPOLLOUT, {u32=1113026480, u64=139729284067248}}, {EPOLLIN, {u32=3, u64=3}}}, 34, 0) = 3
    clock_gettime(CLOCK_BOOTTIME, {1854741, 787993414}) = 0
    read(14, "\1\0\0\0\0\0\0\0", 8)         = 8
    recvmsg(71, 0x7ffcc8bf92e0, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
    timerfd_settime(14, TFD_TIMER_ABSTIME, {it_interval={0, 0}, it_value={1854613, 287736000}}, NULL) = 0
    epoll_wait(4, {{EPOLLOUT, {u32=1112296432, u64=139729283337200}}, {EPOLLOUT, {u32=1113026480, u64=139729284067248}}, {EPOLLIN, {u32=3, u64=3}}}, 34, 0) = 3
    [etc.]

I tried to break on source_dispatch like described in [1], but gdb
couldn't find the function (because I had no debugging symbols?).

Now (just before I wanted to start this mail :D) I got a force-reboot:

    [...]
    Apr 16 08:47:29 tonks systemd[1]: Looping too fast. Throttling execution a little.
    Apr 16 08:47:30 tonks systemd[1]: Job reboot.target/start timed out.
    Apr 16 08:47:30 tonks systemd[1]: Timed out starting Reboot.
    Apr 16 08:47:30 tonks systemd[1]: Job reboot.target/start failed with result 'timeout'.
    Apr 16 08:47:30 tonks systemd[1]: Forcibly rebooting as result of failure.
    Apr 16 08:47:30 tonks systemd[1]: Shutting down.
    Apr 16 08:47:30 tonks systemd[1]: Hardware watchdog 'iTCO_wdt', version 0
    Apr 16 08:47:30 tonks systemd[1]: Set hardware watchdog to 10min.
    Apr 16 08:47:30 tonks kernel: watchdog watchdog0: watchdog did not stop!
    Apr 16 08:47:32 tonks systemd-shutdown[1]: Sending SIGTERM to remaining processes...

I don't remember ever using reboot/reboot.target by hand.

I'm running systemd 218 on Arch Linux.

Not sure if there's anything else I can provide, as it's now gone
after the reboot - but if there is, please let me know!

Florian

[1] http://lists.freedesktop.org/archives/systemd-devel/2015-February/028541.html
[2] http://lists.freedesktop.org/archives/systemd-devel/2014-December/025867.html

-- 
http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
         I love long mails! | http://email.is-not-s.ms/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.freedesktop.org/archives/systemd-devel/attachments/20150416/52ce548a/attachment.sig>