[systemd-devel] Networkd doesn't create route for IP in different but connected net with GatewayOnLink= Inbox

[LunarLambda]

Hello,

I was recently tasked with moving existing network configuration for a
machine and some nspawn containers from iupdown to networkd.

The situation looks as follows:

A single VPS has 3 IPs. One 37.x.x.x/22, and two 91.x.x.x/32. The 37-ip is
to be routed to the main server, whereas the two 91-ips should be routed
directly to nspawn containers running on the server. The server uses
systemd 247 and the container uses systemd 252, both Debian.

I created a bridge netdev like so:

[NetDev]
Name=br0
Type=bridge
# Matches physical network card
MACAddress=AA:BB:CC:DD:EE:FF

Bound the physical ethernet to it like so:

[Match]
Name=ens3

[Network]
Bridge=br0

And set up the main IP for the bridge like so:

[Match]
Name=br0

[Network]
DNS=...
DNS=...
Address=37.x.x.x/22
Gateway=37.x.x.1

The nspawn containers are added to the bridge via

[Network]
Bridge=br0

Up until this point everything works. However, configuring networking
between the host and containers proved quite difficult and I'm unsure
whether I'm doing something wrong or networkd is.

What I tried was the following, inside the container:

[Match]
Virtualization=container
Name=host0

[Address]
Address=91.x.x.x/32

[Route]
Gateway=37.x.x.x
GatewayOnLink=true

However, this did not create any usable routes to the host, nor did it
throw any errors in the journal. Pinging the host does not work.

Manually creating the routes with ip route did work:

ip r add 37.x.x.x dev host0 onlink
ip r add default dev host0 via 37.x.x.x

I tried a variety of different combinations of options in the .network
file, Scope, Type, etc...

The only thing that successfully created any routes was the following:

[Match]
Virtualization=container
Name=host0

[Address]
Address=91.x.x.x/32
Peer=37.x.x.x/32

[Network]
Gateway=37.x.x.x

This strikes me as odd because nowhere in the documentation, nor in any
online searching could I find this described as necessary (beyond the
manpage mentioning that Peer= exists)

On the host side, I thought the bridge device, acting on Layer 2, would
automatically figure out routes to the containers (via ARP), or that nspawn
and networkd would interact in some way to add routes. However, this didn't
seem to happen, so I also had to add the following to the bridge's .network
file:

[Route]
Source=37.x.x.x
Destination=91.x.x.A

[Route]
Source=37.x.x.x
Destination=91.x.x.B

With all of this, everything works fine now. However, since the routes,
both host-to-container and container-to-host, differ somewhat from the old
(also working) setup, and some of the steps necessary I could not find
described anywhere, I am left wondering if I fundamentally misunderstand
something about how Linux networking works, or if perhaps networkd is
behaving oddly because of the IP addresses being considered in different
networks.

I would love to find a conclusive answer to this, especially because I want
to make sure I understood the fundamental concepts involved correctly.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20230712/c9289484/attachment-0001.htm>