[systemd-devel] nspawn+networkd sometimes fail to configure IPv6 properly

[Kevin P. Fleming]

I've got three systems which host nspawn-based containers, using
networkd for network configuration on both the host and inside the
containers. All of the systems are running Debian systemd packages
(some version 241 (buster) and some 247.2 (bullseye)). The behavior
has been seen with kernels 5.4, 5.9, and 5.10 (both Debian kernel
packages and a hand-built vanilla kernel package). There are no
firewalls in use.

An example configuration:

host - /etc/systemd/nspawn/mqtt20.nspawn
----
[Files]
PrivateUsersChown=yes
[Network]
VirtualEthernetExtra=mqtt20:srv

host - /etc/systemd/network/mqtt20.network
---
[Match]
Name=mqtt20

[Network]
Address=192.168.254.108/32
Address=fd80:ae6b:5f43:254::108/128

[Route]
Destination=192.168.64.108
Scope=link

[Route]
Destination=2001:470:8afe:64::108

[Route]
Destination=fd80:ae6b:5f43:64::108

container - /etc/systemd/network/primary.network
---
[Match]
Name=srv

[Network]
Address=192.168.64.108/32
Address=2001:470:8afe:64::108/128
Address=fd80:ae6b:5f43:64::108/128
DNS=fd80:ae6b:5f43:1::8

[Route]
Destination=192.168.254.108/32
Scope=link

[Route]
Gateway=192.168.254.108
Destination=0.0.0.0/0

[Route]
Destination=fd80:ae6b:5f43:254::108/128

[Route]
Gateway=fd80:ae6b:5f43:254::108
Destination=::/0

---

Layer 3 networking is used, on virtual Ethernet devices.

Sometimes, after a system startup, some of the containers are not
reachable over IPv6. When this happens, their IPv4 connectivity is
fine. Running 'machinectl stop <c>' followed by 'machinectl start <c>'
always cures the problem.

When a container is in this state, 'ip link ls' and 'ip addr ls' on
the host and in the container don't display anything out of the
ordinary (the details match those of another container on the same
host which works properly).

I've run tcpdump on the veth device for a broken container and then
sent ICMPv6 pings from another system on the network; what I see is
that the host sends IPv6 Neighbor Solicitation requests on the veth
device to discover the container's layer 2 address, but no replies are
sent. Running tcpdump inside the container shows the same thing; NS is
received, but no reply is sent.

It's as if the IPv6 stack in the container's network namespace is
just... not listening at all.

Can anyone suggest ways to troubleshoot this beyond the simple things
I've listed above?