[systemd-devel] systemd-networkd packet loss Ubuntu 18.04 (systemd 237)

[Paul Menzel]

Dear Luke,


On 2020-07-08 13:10, Luke Alexander wrote:

> I had a look through some of the issues against the systemd github repo but
> could not find any matching - there were a couple which look promising but
> don't completely resolve our issue.
> 
> Our issue is that we have a k8s (1.18, kube-router CNI) cluster comprised
> of a number of Ubuntu 16.04 nodes, we are in the process if upgrading the
> nodes to Ubuntu 18.04 - however after upgrading the first node and adding
> it back to the cluster we observed high packet loss from other cluster
> nodes to the 18.04 node as well as ping error messages when running a
> continuous ping from the 18.04 node to another node in the cluster, eg:
> 
> 64 bytes from 10.8.11.1: icmp_seq=91 ttl=64 time=0.088 ms
> 64 bytes from 10.8.11.1: icmp_seq=92 ttl=64 time=0.076 ms
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> 
> Another observation is that one of our daemonset pods (promtail) could not
> start.
> 
> systemctl --version
> systemd 237
> +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP
> +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN
> -PCRE2 default-hierarchy=hybrid
> 
> After many days of troubleshooting various possibilities, changing network
> cable, NIC, a different server, various sysctl values, I eventually tried
> switching from systemd-networkd to NetworkManager as the backend via
> netplan config. After rebooting the server with NetworkManager in place and
> re-adding the node back to the k8s cluster, the packet loss disappeared as
> did the ping errors and the daemonset pod (promtail) also started normally.
> 
> I would like to use Ubuntu's default of systemd-networkd to handle our
> NIC/route config - but cannot until I've figured out what in
> systemd-networkd is breaking.
> 
> Any help is much appreciated on this matter.

Unfortunately, upstream only supports the last two upstream versions of
systemd. Please report the issue to the Ubuntu bug tracker Launchpad [1].
It would probably help to verify, if the issue is still present in the newest
version. So maybe try Ubuntu 20.04 first to see if it’s fixed there.
Otherwise, building systemd yourself in Ubuntu should be doable too.

I am sorry for the unsatisfying answer.


Kind regards,

Paul


[1]: https://bugs.launchpad.net/ubuntu/+source/systemd

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5174 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20200708/4c5b3c94/attachment-0001.bin>