[systemd-devel] help re-configuring bond and ipoib devices/networks

Tue Jul 30 04:09:02 UTC 2024

Some network types use longer or shorter addresses, not all of them try to
mimic Ethernet.

For example FireWire uses 64-bit hardware addresses but IP-over-FW extends
it to 128-bit addresses in ARP for technical reasons, and I think it's the
same for Infiniband and IPoIB.

Unfortunately Networkd doesn't understand any of it.

On Tue, Jul 30, 2024, 04:42 serenissi <serenissi at inventati.org> wrote:

> I can't tell more about the IPoIB going down after networkd restart
> without additional debugging info. But from the complains, did you try
> removing the problematic keys (ipoib is part of netdev, not network.
> network has no knowledge of the device type)?
>
> Also are you sure 80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11
> is the mac address? It is clearly more than 48 bits.
> On 7/29/24 08:43, Chandler Sobel-Sorenson wrote:
>
>     I'm quite frustrated having spent many hours and little success, when
> things were perfectly fine before our backup generator decided not to kick
> in and power surges ensued, messing up all kinds of things stupid
> electrons.  We have a server that is a bit more important than the others,
> runs LDAP and hosts our home directories, and runs the subnet manager for
> an Infiniband network.
>
>     The main uplink is a 2x10gbe bond device, and there's a local 1gbe
> 10-net interface/network, and an Infiniband with IPoIB.  The main issue is
> getting the IPoIB working.  The bond and its 2 links work fine after the
> machine boots, but whenever I restart systemd-networkd, it becomes
> unreachable even though there aren't any problems reported in the system
> logs and networkctl reports it's still routable.  The only thing I've
> figured to get it reachable again is to reboot the system, so I'd like to
> figure out the problem there because that shouldn't be happening.
>
>     The IPoIB is driving me nuts, I read through all of systemd.network
> and systemd.netdev docs, and it's somehow become rogue and unmanaged when
> it used to be configured.  systemd-networkd keeps telling me about unknown
> keys and unknown sections, even though they were all added in versions
> previous to the current, which is:
>
> systemd 252 (252.26-1~deb12u2) running on Debian 11 Linux 5.10.0-20-amd64
>
> Below are the configs and messages from systemd.  Hope you can help.
> # networkctl
> *IDX LINK     TYPE       OPERATIONAL SETUP     *
>   1 lo       loopback   carrier     unmanaged
>   2 ens1f0   ether      enslaved    configured <--10gbe link 1
>   3 enp4s0f0 ether      off         unmanaged  <--not connected
>   4 ens1f1   ether      enslaved    configured <--10gbe link 2
>   5 enp4s0f1 ether      routable    configured <--1gbe local 10-net
>   6 ibs3     infiniband off         unmanaged  <--lazy, rogue
>   7 ibs3d1   infiniband off         unmanaged  <--not connected
>   8 bond007  bond       routable    configured
>
> *Infiniband, IPoIB, ibs3 * # cat 10-ibs3.netdev
> [Match]
>
> [NetDev]
> Name=ibs3
> Kind=ipoib
> MTUBytes=65520
> MACAddress=80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11
>
> [IPoIB]
> Mode=connected
>
>
> # cat ibs3-10_10_11_203.network
> [Match]
> Name=ibs3
>
> PermanentMACAddress=80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11
> Path=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/ibs3
> Driver=ib_ipoib
> Type=infiniband
> Kind=ipoib
> Property=ID_NET_MANAGED_BY=io.systemd.Network
>
> [Link]
> MTUBytes=65520
>
> [Network]
> Kind=ipoib
> Address=10.10.11.203/24
> Gateway=10.10.11.203
> LinkLocalAddressing=no
> IPv4AcceptLocal=yes
> KeepConfiguration=static
>
> [IPoIB]
> Mode=connected
> systemd-networkd complaints
> /etc/systemd/network/10-ibs3.netdev:8: Not a valid hardware address,
> ignoring assignment:
> 80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11
> /etc/systemd/network/10-ibs3.netdev:8: Not a valid hardware address,
> ignoring assignment:
> 80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11
> /etc/systemd/network/ibs3-10_10_11_203.network:15: Unknown key 'Kind' in
> section [Network], ignoring.
> /etc/systemd/network/ibs3-10_10_11_203.network:22: Unknown section
> 'IPoIB'. Ignoring.
>
> *bond*
> # cat 10-bond007.netdev
> [NetDev]
> Name=bond007
> Kind=bond
> MTUBytes=9000
>
> [Bond]
> Mode=802.3ad
> MIIMonitorSec=1000
> UpDelaySec=1000
> DownDelaySec=2000
>
>
> # cat Intel_X710_DA2-bond007.network
> [Match]
> Path=pci-0000:05:00.0
> Path=pci-0000:05:00.1
>
> [Network]
> Bond=bond007
>
>
> # cat bond007-10_140_78_70.network
> [Match]
> Name=bond007
>
> [Network]
> Address=10.140.78.70/28
> Gateway=10.140.78.65
> DNS=128.196.11.233
> DNS=128.196.11.234
> DNS=128.196.11.235
> LinkLocalAddressing=no
> IPv6AcceptRA=no
>
> Best Regards,
> Chandler
> [image: The University of Arizona block 'A' logo.]
> *Chandler Sobel-Sorenson*
> Sr. Systems Administrator
> Arizona Genomics Institute
> School of Plant Sciences—Research
> THE UNIVERSITY OF ARIZONA
>
> Thomas W. Keating Bioresearch Bldg. | Rm. 200A24
> 1657 E. Helen St. | Tucson, AZ 85721
> <https://www.google.com/maps/search/1657+E.+Helen+St.+%7C+Tucson,+AZ+85721+%0D%0A++++++++++++++++++++++Office:++520?entry=gmail&source=g>
> Office:
> <https://www.google.com/maps/search/1657+E.+Helen+St.+%7C+Tucson,+AZ+85721+%0D%0A++++++++++++++++++++++Office:++520?entry=gmail&source=g>
> 520-626-9589 | Cell: 520-907-4352
>
> chandler at genome.arizona.edu <+chandler at genome.arizona.edu>
> Pronouns: he/him/his
> *www.genome.arizona.edu* <https://www.genome.arizona.edu/>
>
> [image: Integrity, Compassion, Exploration, Adaptation, Inclusion,
> Determination] <https://brand.arizona.edu/signature>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20240730/61dab46b/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ua-sig-logo.png
Type: image/png
Size: 14173 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20240730/61dab46b/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature-footer.png
Type: image/png
Size: 24654 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20240730/61dab46b/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature-footer.png
Type: image/png
Size: 24654 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20240730/61dab46b/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ua-sig-logo.png
Type: image/png
Size: 14173 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20240730/61dab46b/attachment-0007.png>