[systemd-devel] help re-configuring bond and ipoib devices/networks

Mon Jul 29 03:13:30 UTC 2024

    I'm quite frustrated having spent many hours and little success,
when things were perfectly fine before our backup generator decided not
to kick in and power surges ensued, messing up all kinds of things
stupid electrons.  We have a server that is a bit more important than
the others, runs LDAP and hosts our home directories, and runs the
subnet manager for an Infiniband network.

    The main uplink is a 2x10gbe bond device, and there's a local 1gbe
10-net interface/network, and an Infiniband with IPoIB.  The main issue
is getting the IPoIB working.  The bond and its 2 links work fine after
the machine boots, but whenever I restart systemd-networkd, it becomes
unreachable even though there aren't any problems reported in the system
logs and networkctl reports it's still routable.  The only thing I've
figured to get it reachable again is to reboot the system, so I'd like
to figure out the problem there because that shouldn't be happening.

    The IPoIB is driving me nuts, I read through all of systemd.network
and systemd.netdev docs, and it's somehow become rogue and unmanaged
when it used to be configured.  systemd-networkd keeps telling me about
unknown keys and unknown sections, even though they were all added in
versions previous to the current, which is:

systemd 252 (252.26-1~deb12u2) running on Debian 11 Linux 5.10.0-20-amd64

Below are the configs and messages from systemd.  Hope you can help.

# networkctl
_IDX LINK     TYPE       OPERATIONAL SETUP     _
  1 lo       loopback   carrier     unmanaged
  2 ens1f0   ether      enslaved    configured <--10gbe link 1
  3 enp4s0f0 ether      off         unmanaged  <--not connected
  4 ens1f1   ether      enslaved    configured <--10gbe link 2
  5 enp4s0f1 ether      routable    configured <--1gbe local 10-net
  6 ibs3     infiniband off         unmanaged  <--lazy, rogue
  7 ibs3d1   infiniband off         unmanaged  <--not connected
  8 bond007  bond       routable    configured

      _Infiniband, IPoIB, ibs3
      _

# cat 10-ibs3.netdev
[Match]

[NetDev]
Name=ibs3
Kind=ipoib
MTUBytes=65520
MACAddress=80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11

[IPoIB]
Mode=connected

# cat ibs3-10_10_11_203.network
[Match]
Name=ibs3
PermanentMACAddress=80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11
Path=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/ibs3
Driver=ib_ipoib
Type=infiniband
Kind=ipoib
Property=ID_NET_MANAGED_BY=io.systemd.Network

[Link]
MTUBytes=65520

[Network]
Kind=ipoib
Address=10.10.11.203/24
Gateway=10.10.11.203
LinkLocalAddressing=no
IPv4AcceptLocal=yes
KeepConfiguration=static

[IPoIB]
Mode=connected

      systemd-networkd complaints

/etc/systemd/network/10-ibs3.netdev:8: Not a valid hardware address,
ignoring assignment:
80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11
/etc/systemd/network/10-ibs3.netdev:8: Not a valid hardware address,
ignoring assignment:
80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11
/etc/systemd/network/ibs3-10_10_11_203.network:15: Unknown key 'Kind' in
section [Network], ignoring.
/etc/systemd/network/ibs3-10_10_11_203.network:22: Unknown section
'IPoIB'. Ignoring.

      _bond_

# cat 10-bond007.netdev
[NetDev]
Name=bond007
Kind=bond
MTUBytes=9000

[Bond]
Mode=802.3ad
MIIMonitorSec=1000
UpDelaySec=1000
DownDelaySec=2000

# cat Intel_X710_DA2-bond007.network
[Match]
Path=pci-0000:05:00.0
Path=pci-0000:05:00.1

[Network]
Bond=bond007

# cat bond007-10_140_78_70.network
[Match]
Name=bond007

[Network]
Address=10.140.78.70/28
Gateway=10.140.78.65
DNS=128.196.11.233
DNS=128.196.11.234
DNS=128.196.11.235
LinkLocalAddressing=no
IPv6AcceptRA=no

Best Regards,
Chandler

The University of Arizona block 'A' logo.
	*Chandler Sobel-Sorenson*
Sr. Systems Administrator
Arizona Genomics Institute
School of Plant Sciences—Research
THE UNIVERSITY OF ARIZONA

Thomas W. Keating Bioresearch Bldg. | Rm. 200A24
1657 E. Helen St. | Tucson, AZ 85721
Office: 520-626-9589 <tel:520-626-9589> | Cell: 520-907-4352
<tel:520-907-4352>

chandler at genome.arizona.edu <mailto: chandler at genome.arizona.edu>
Pronouns: he/him/his
*www.genome.arizona.edu* <https://www.genome.arizona.edu/>

	Integrity, Compassion, Exploration, Adaptation, Inclusion,
Determination <https://brand.arizona.edu/signature>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20240728/c81f197f/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ua-sig-logo.png
Type: image/png
Size: 14173 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20240728/c81f197f/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature-footer.png
Type: image/png
Size: 24654 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20240728/c81f197f/attachment-0003.png>