[systemd-devel] nspawn+networkd sometimes fail to configure IPv6 properly

[Kevin P. Fleming]

While debugging a different problem (which I'll report in an issue in
the GitHub repo), I found something which appears to be the underlying
cause.

I rebooted a machine which has five nspawn containers, all configured
identically for networking (except for addresses, of course).
Immediately after the boot, 'machinectl list' shows all five
containers running, and all five are accessible over IPv4.

However, only two are accessible over IPv6, and:

---
root at net20:~# ip neigh show proxy
2607:5300:203:5215::1 dev ve-mail20  proxy
2607:5300:203:5215::1 dev ve-diw20  proxy
---

Only two are listed with proxy addresses for the host on their veth
devices; these are the two that are accessible.

---
root at net20:~# machinectl stop ns3
root at net20:~# machinectl start ns3
root at net20:~# ip neigh show proxy
2607:5300:203:5215::1 dev ve-ns3  proxy
2607:5300:203:5215::1 dev ve-mail20  proxy
2607:5300:203:5215::1 dev ve-diw20  proxy
---

After a stop/start of the 'ns3' container, it now has the necessary
proxy address on its veth device, and it is accessible over IPv6.

---
root at net20:~# machinectl stop matrix20
root at net20:~# machinectl start matrix20
root at net20:~# ip neigh show proxy
2607:5300:203:5215::1 dev ve-matrix20  proxy
2607:5300:203:5215::1 dev ve-ns3  proxy
2607:5300:203:5215::1 dev ve-mail20  proxy
2607:5300:203:5215::1 dev ve-diw20  proxy
root at net20:~# machinectl stop ldl20
root at net20:~# machinectl start ldl20
root at net20:~# ip neigh show proxy
2607:5300:203:5215::1 dev ve-ldl20  proxy
2607:5300:203:5215::1 dev ve-matrix20  proxy
2607:5300:203:5215::1 dev ve-ns3  proxy
2607:5300:203:5215::1 dev ve-mail20  proxy
2607:5300:203:5215::1 dev ve-diw20  proxy
---

Same results for stop/start of the remaining two containers; they get
the necessary proxy address, and become reachable.

I've got DEBUG logging enabled for systemd-networkd but there is no
evidence of any problem related to proxy address addition
(https://gist.github.com/kpfleming/4b6c721edaf9d2ff4a24dadbb002fb0c).

I think the next step is going to be building systemd from source to
verify that the Debian 247.2 packages aren't at fault, and then adding
more debugging output in the necessary places.

On Sun, Jan 24, 2021 at 10:14 AM Kevin P. Fleming <kevin at km6g.us> wrote:
>
> I've got three systems which host nspawn-based containers, using
> networkd for network configuration on both the host and inside the
> containers. All of the systems are running Debian systemd packages
> (some version 241 (buster) and some 247.2 (bullseye)). The behavior
> has been seen with kernels 5.4, 5.9, and 5.10 (both Debian kernel
> packages and a hand-built vanilla kernel package). There are no
> firewalls in use.
>
> An example configuration:
>
> host - /etc/systemd/nspawn/mqtt20.nspawn
> ----
> [Files]
> PrivateUsersChown=yes
> [Network]
> VirtualEthernetExtra=mqtt20:srv
>
> host - /etc/systemd/network/mqtt20.network
> ---
> [Match]
> Name=mqtt20
>
> [Network]
> Address=192.168.254.108/32
> Address=fd80:ae6b:5f43:254::108/128
>
> [Route]
> Destination=192.168.64.108
> Scope=link
>
> [Route]
> Destination=2001:470:8afe:64::108
>
> [Route]
> Destination=fd80:ae6b:5f43:64::108
>
> container - /etc/systemd/network/primary.network
> ---
> [Match]
> Name=srv
>
> [Network]
> Address=192.168.64.108/32
> Address=2001:470:8afe:64::108/128
> Address=fd80:ae6b:5f43:64::108/128
> DNS=fd80:ae6b:5f43:1::8
>
> [Route]
> Destination=192.168.254.108/32
> Scope=link
>
> [Route]
> Gateway=192.168.254.108
> Destination=0.0.0.0/0
>
> [Route]
> Destination=fd80:ae6b:5f43:254::108/128
>
> [Route]
> Gateway=fd80:ae6b:5f43:254::108
> Destination=::/0
>
> ---
>
> Layer 3 networking is used, on virtual Ethernet devices.
>
> Sometimes, after a system startup, some of the containers are not
> reachable over IPv6. When this happens, their IPv4 connectivity is
> fine. Running 'machinectl stop <c>' followed by 'machinectl start <c>'
> always cures the problem.
>
> When a container is in this state, 'ip link ls' and 'ip addr ls' on
> the host and in the container don't display anything out of the
> ordinary (the details match those of another container on the same
> host which works properly).
>
> I've run tcpdump on the veth device for a broken container and then
> sent ICMPv6 pings from another system on the network; what I see is
> that the host sends IPv6 Neighbor Solicitation requests on the veth
> device to discover the container's layer 2 address, but no replies are
> sent. Running tcpdump inside the container shows the same thing; NS is
> received, but no reply is sent.
>
> It's as if the IPv6 stack in the container's network namespace is
> just... not listening at all.
>
> Can anyone suggest ways to troubleshoot this beyond the simple things
> I've listed above?