[systemd-devel] systemd-timesyncd dies on one machine but runs okay on another

Mon Sep 3 12:40:50 UTC 2018

Hallo all!

I'm working on an embedded project consisting of an host and numerous virtual machines and am facing problems related to time management. systemd-timesyncd dies on one machine (HOST), but runs okay on another (VM2) with identical config.

The project setup is as follows:

HOST: Runs a (heavily) modified Debian with systemd, retrieves time via NTP from VM3 using systemd-timesyncd.
* VM1: Runs openwrt and serves time via NTP using chrony
* VM2: Runs a (heavily) modified Debian with SELINUX and systemd, retrieves time via NTP from VM3 using systemd-timesyncd
* VM3-?: Some other VMs which seem to run fine

The symptoms are as follows:

HOST: `systemd-timesyncd.service` is reported as inactive (dead). A time update (adjusting the time by 2 days) took place. Then, after 5min, the service died. (It always seems to die 5-15min after a time update.)
```
root at HOST:/var/log# systemctl status systemd-timesyncd
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: inactive (dead) since Wed 2018-09-05 07:30:02 CEST; 1 day 17h left
     Docs: man:systemd-timesyncd.service(8)
  Process: 4724 ExecStart=/lib/systemd/systemd-timesyncd (code=exited, status=0/SUCCESS)
 Main PID: 4724 (code=exited, status=0/SUCCESS)
   Status: "Idle."

Sep 03 11:51:58 konnektor systemd[1]: Starting Network Time Synchronization...
Sep 03 11:51:58 konnektor systemd[1]: Started Network Time Synchronization.
Sep 05 07:25:31 konnektor systemd-timesyncd[4724]: Synchronized to time server 192.168.253.1:123 (192.168.253.1).
Sep 05 07:30:02 konnektor systemd[1]: Stopping Network Time Synchronization...
Sep 05 07:30:02 konnektor systemd[1]: Stopped Network Time Synchronization.
```

VM2:`systemd-timesyncd.service` is reported to be active (running), time updates occur regularly.
```
root at VM2:~# systemctl status systemd-timesyncd
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: active (running) since Thu 2018-08-30 20:53:19 CEST; 5 days ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 12125 (systemd-timesyn)
   Status: "Synchronized to time server 192.168.253.1:123 (192.168.253.1)."
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/systemd-timesyncd.service
           └─12125 /lib/systemd/systemd-timesyncd

Aug 30 20:53:19 k-basisdienste systemd[1]: Starting Network Time Synchronization...
Aug 30 20:53:19 k-basisdienste systemd[1]: Started Network Time Synchronization.
Aug 30 20:44:53 k-basisdienste systemd-timesyncd[12125]: Synchronized to time server 192.168.253.1:123 (192.168.253.1).
```
On this VM, we DO have the problem of the time jumping by 15s every 50s or so. I think is caused by the time server (chrony) over-compensating a clockdrift. Hence, I think this is totally unrelated. I just mention it for the sake of completeness.

When I query chrony on VM1 for statistics on the connected clients, it as well reports that the other VMs are updated every ~50s, while the last update to the host system occured hours ago.
```
root at VM1:~# chronyc clients
Hostname                      NTP   Drop Int IntL Last     Cmd   Drop Int  Last
===============================================================================
192.168.253.14              11054      0   6   -     1       0      0   -     -
192.168.253.10              11053      0   6   -     1       0      0   -     -
192.168.253.2               11053      0   6   -     1       0      0   -     -
192.168.253.18              11053      0   6   -     1       0      0   -     -
192.168.253.3                 390      0  10   -  237m       0      0   -     -  COMMENT: This is the host machine
```

systemd-timesyncd on both HOST and VM2 are using an identical unit file and also timesyncd.conf (Please find both below.)

Questions I'm asking myself are:
* What could be the reason for systemd-timesyncd dying on the host machine? How can I find details?
* If I interpret the output correctly, systemd-timesyncd died with an exit code of 0 (success). How is that possible? Shouldn't it remain active, just the way it does on VM2?

Thanks a lot for your support!
Manuel

systemd-timesyncd configuration for both HOST and VM2
```
root at HOST, VM2:/var/log# cat /lib/systemd/system/systemd-networkd.service
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Network Service
Documentation=man:systemd-networkd.service(8)
ConditionCapability=CAP_NET_ADMIN
DefaultDependencies=no
# systemd-udevd.service can be dropped once tuntap is moved to netlink
After=systemd-udevd.service network-pre.target systemd-sysusers.service systemd-sysctl.service
Before=network.target multi-user.target shutdown.target
Conflicts=shutdown.target
Wants=network.target

# On kdbus systems we pull in the busname explicitly, because it
# carries policy that allows the daemon to acquire its name.
Wants=org.freedesktop.network1.busname
After=org.freedesktop.network1.busname

[Service]
Type=notify
Restart=on-failure
RestartSec=0
ExecStart=/lib/systemd/systemd-networkd
WatchdogSec=3min
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_BROADCAST CAP_NET_RAW CAP_SETUID CAP_SETGID CAP_SETPCAP CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER
ProtectSystem=full
ProtectHome=yes
ProtectControlGroups=yes
MemoryDenyWriteExecute=yes
RestrictRealtime=yes
RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6 AF_PACKET
SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount @obsolete @raw-io

[Install]
WantedBy=multi-user.target
Also=systemd-networkd.socket

root at HOST, VM2:/var/log# cat /etc/systemd/timesyncd.conf 
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.

[Time]
NTP=192.168.253.1   COMMENT: This is the IP of VM1
```