[systemd-devel] help re-configuring bond and ipoib devices/networks

Tue Jul 30 01:35:09 UTC 2024

I can't tell more about the IPoIB going down after networkd restart 
without additional debugging info. But from the complains, did you try 
removing the problematic keys (ipoib is part of netdev, not network. 
network has no knowledge of the device type)?

Also are you sure 
80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11 is the mac 
address? It is clearly more than 48 bits.

On 7/29/24 08:43, Chandler Sobel-Sorenson wrote:
>
>     I'm quite frustrated having spent many hours and little success, 
> when things were perfectly fine before our backup generator decided 
> not to kick in and power surges ensued, messing up all kinds of things 
> stupid electrons.  We have a server that is a bit more important than 
> the others, runs LDAP and hosts our home directories, and runs the 
> subnet manager for an Infiniband network.
>
>     The main uplink is a 2x10gbe bond device, and there's a local 1gbe 
> 10-net interface/network, and an Infiniband with IPoIB.  The main 
> issue is getting the IPoIB working.  The bond and its 2 links work 
> fine after the machine boots, but whenever I restart systemd-networkd, 
> it becomes unreachable even though there aren't any problems reported 
> in the system logs and networkctl reports it's still routable.  The 
> only thing I've figured to get it reachable again is to reboot the 
> system, so I'd like to figure out the problem there because that 
> shouldn't be happening.
>
>     The IPoIB is driving me nuts, I read through all of 
> systemd.network and systemd.netdev docs, and it's somehow become rogue 
> and unmanaged when it used to be configured. systemd-networkd keeps 
> telling me about unknown keys and unknown sections, even though they 
> were all added in versions previous to the current, which is:
>
> systemd 252 (252.26-1~deb12u2) running on Debian 11 Linux 5.10.0-20-amd64
>
> Below are the configs and messages from systemd.  Hope you can help.
>
> # networkctl
> _IDX LINK     TYPE       OPERATIONAL SETUP _
>   1 lo       loopback carrier unmanaged
>   2 ens1f0   ether enslaved configured <--10gbe link 1
>   3 enp4s0f0 ether      off         unmanaged <--not connected
>   4 ens1f1   ether enslaved configured <--10gbe link 2
>   5 enp4s0f1 ether routable configured <--1gbe local 10-net
>   6 ibs3     infiniband off         unmanaged <--lazy, rogue
>   7 ibs3d1   infiniband off         unmanaged <--not connected
>   8 bond007  bond routable configured
>
>
>       _Infiniband, IPoIB, ibs3
>       _
>
> # cat 10-ibs3.netdev
> [Match]
>
> [NetDev]
> Name=ibs3
> Kind=ipoib
> MTUBytes=65520
> MACAddress=80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11
>
> [IPoIB]
> Mode=connected
>
>
> # cat ibs3-10_10_11_203.network
> [Match]
> Name=ibs3
> PermanentMACAddress=80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11
> Path=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/ibs3
> Driver=ib_ipoib
> Type=infiniband
> Kind=ipoib
> Property=ID_NET_MANAGED_BY=io.systemd.Network
>
> [Link]
> MTUBytes=65520
>
> [Network]
> Kind=ipoib
> Address=10.10.11.203/24
> Gateway=10.10.11.203
> LinkLocalAddressing=no
> IPv4AcceptLocal=yes
> KeepConfiguration=static
>
> [IPoIB]
> Mode=connected
>
>
>       systemd-networkd complaints
>
> /etc/systemd/network/10-ibs3.netdev:8: Not a valid hardware address, 
> ignoring assignment: 
> 80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11
> /etc/systemd/network/10-ibs3.netdev:8: Not a valid hardware address, 
> ignoring assignment: 
> 80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11
> /etc/systemd/network/ibs3-10_10_11_203.network:15: Unknown key 'Kind' 
> in section [Network], ignoring.
> /etc/systemd/network/ibs3-10_10_11_203.network:22: Unknown section 
> 'IPoIB'. Ignoring.
>
>
>       _bond_
>
> # cat 10-bond007.netdev
> [NetDev]
> Name=bond007
> Kind=bond
> MTUBytes=9000
>
> [Bond]
> Mode=802.3ad
> MIIMonitorSec=1000
> UpDelaySec=1000
> DownDelaySec=2000
>
>
> # cat Intel_X710_DA2-bond007.network
> [Match]
> Path=pci-0000:05:00.0
> Path=pci-0000:05:00.1
>
> [Network]
> Bond=bond007
>
>
> # cat bond007-10_140_78_70.network
> [Match]
> Name=bond007
>
> [Network]
> Address=10.140.78.70/28
> Gateway=10.140.78.65
> DNS=128.196.11.233
> DNS=128.196.11.234
> DNS=128.196.11.235
> LinkLocalAddressing=no
> IPv6AcceptRA=no
>
> Best Regards,
> Chandler
>
> The University of Arizona block 'A' logo.
> 	*Chandler Sobel-Sorenson*
> Sr. Systems Administrator
> Arizona Genomics Institute
> School of Plant Sciences—Research
> THE UNIVERSITY OF ARIZONA
>
> Thomas W. Keating Bioresearch Bldg. | Rm. 200A24
> 1657 E. Helen St. | Tucson, AZ 85721
> Office: 520-626-9589 <tel:520-626-9589> | Cell: 520-907-4352 
> <tel:520-907-4352>
>
> chandler at genome.arizona.edu <mailto: chandler at genome.arizona.edu>
> Pronouns: he/him/his
> *www.genome.arizona.edu* <https://www.genome.arizona.edu/>
>
> 	Integrity, Compassion, Exploration, Adaptation, Inclusion, 
> Determination <https://brand.arizona.edu/signature>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20240730/b02b42fe/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ua-sig-logo.png
Type: image/png
Size: 14173 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20240730/b02b42fe/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature-footer.png
Type: image/png
Size: 24654 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20240730/b02b42fe/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0x20257A7131FFF28B.asc
Type: application/pgp-keys
Size: 652 bytes
Desc: OpenPGP public key
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20240730/b02b42fe/attachment-0001.key>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 236 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20240730/b02b42fe/attachment-0001.sig>