<div dir="auto">Some network types use longer or shorter addresses, not all of them try to mimic Ethernet.<div dir="auto"> </div><div dir="auto">For example FireWire uses 64-bit hardware addresses but IP-over-FW extends it to 128-bit addresses in ARP for technical reasons, and I think it's the same for Infiniband and IPoIB.</div><div dir="auto"> </div><div dir="auto">Unfortunately Networkd doesn't understand any of it.</div></div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 30, 2024, 04:42 serenissi <<a href="mailto:serenissi@inventati.org">serenissi@inventati.org</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div> I can't tell more about the IPoIB going down after networkd restart without additional debugging info. But from the complains, did you try removing the problematic keys (ipoib is part of netdev, not network. network has no knowledge of the device type)? Also are you sure <tt>80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11 is the mac address? It is clearly more than 48 bits. </tt> <div>On 7/29/24 08:43, Chandler Sobel-Sorenson wrote: </div> <blockquote type="cite"> I'm quite frustrated having spent many hours and little success, when things were perfectly fine before our backup generator decided not to kick in and power surges ensued, messing up all kinds of things stupid electrons. We have a server that is a bit more important than the others, runs LDAP and hosts our home directories, and runs the subnet manager for an Infiniband network. The main uplink is a 2x10gbe bond device, and there's a local 1gbe 10-net interface/network, and an Infiniband with IPoIB. The main issue is getting the IPoIB working. The bond and its 2 links work fine after the machine boots, but whenever I restart systemd-networkd, it becomes unreachable even though there aren't any problems reported in the system logs and <tt>networkctl</tt> reports it's still routable. The only thing I've figured to get it reachable again is to reboot the system, so I'd like to figure out the problem there because that shouldn't be happening. The IPoIB is driving me nuts, I read through all of systemd.network and systemd.netdev docs, and it's somehow become rogue and unmanaged when it used to be configured. systemd-networkd keeps telling me about unknown keys and unknown sections, even though they were all added in versions previous to the current, which is: systemd 252 (252.26-1~deb12u2) running on Debian 11 Linux 5.10.0-20-amd64 Below are the configs and messages from systemd. Hope you can help. <tt># networkctl IDX LINK TYPE OPERATIONAL SETUP <strike> 1 lo loopback carrier unmanaged</strike> 2 ens1f0 ether enslaved configured </tt><--10gbe link 1<tt> <strike> 3 enp4s0f0 ether off unmanaged</strike> </tt><--not connected <tt> 4 ens1f1 ether enslaved configured </tt><--10gbe link 2 <tt> 5 enp4s0f1 ether routable configured </tt><--1gbe local 10-net <tt> 6 ibs3 infiniband off unmanaged </tt><--lazy, rogue <tt><strike> 7 ibs3d1 infiniband off unmanaged</strike> </tt><--not connected <tt> 8 bond007 bond routable configured</tt> <h3>Infiniband, IPoIB, ibs3<tt> </tt></h3> <tt># cat 10-ibs3.netdev [Match] [NetDev] Name=ibs3 Kind=ipoib MTUBytes=65520 MACAddress=80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11 [IPoIB] Mode=connected # cat ibs3-10_10_11_203.network [Match] Name=ibs3 PermanentMACAddress=80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11 Path=/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/ibs3 Driver=ib_ipoib Type=infiniband Kind=ipoib Property=ID_NET_MANAGED_BY=io.systemd.Network [Link] MTUBytes=65520 [Network] Kind=ipoib Address=<a href="http://10.10.11.203/24" target="_blank" rel="noreferrer">10.10.11.203/24</a> Gateway=10.10.11.203 LinkLocalAddressing=no IPv4AcceptLocal=yes KeepConfiguration=static [IPoIB] Mode=connected </tt> <h3>systemd-networkd complaints </h3> /etc/systemd/network/10-ibs3.netdev:8: Not a valid hardware address, ignoring assignment: 80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11 /etc/systemd/network/10-ibs3.netdev:8: Not a valid hardware address, ignoring assignment: 80:00:02:08:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:6f:85:11 /etc/systemd/network/ibs3-10_10_11_203.network:15: Unknown key 'Kind' in section [Network], ignoring. /etc/systemd/network/ibs3-10_10_11_203.network:22: Unknown section 'IPoIB'. Ignoring. <tt> </tt> <h3>bond </h3> <tt># cat 10-bond007.netdev [NetDev] Name=bond007 Kind=bond MTUBytes=9000 [Bond] Mode=802.3ad MIIMonitorSec=1000 UpDelaySec=1000 DownDelaySec=2000 # cat Intel_X710_DA2-bond007.network [Match] Path=pci-0000:05:00.0 Path=pci-0000:05:00.1 [Network] Bond=bond007 # cat bond007-10_140_78_70.network [Match] Name=bond007 [Network] Address=<a href="http://10.140.78.70/28" target="_blank" rel="noreferrer">10.140.78.70/28</a> Gateway=10.140.78.65 DNS=128.196.11.233 DNS=128.196.11.234 DNS=128.196.11.235 LinkLocalAddressing=no IPv6AcceptRA=no</tt> <div> Best Regards, Chandler </div> <div> <div id="m_6767437406767468510Chandler's signature"> <table> <tbody> <tr> <td style="padding:0px" valign="top"> <img alt="The University of Arizona block 'A' logo." src="cid:part1.PzVkZRSh.0MUoPi5G@inventati.org" width="120"> </td> <td style="padding:0px" valign="top"> Chandler Sobel-Sorenson Sr. Systems Administrator Arizona Genomics Institute School of Plant Sciences—Research THE UNIVERSITY OF ARIZONA Thomas W. Keating Bioresearch Bldg. | Rm. 200A24 <a href="https://www.google.com/maps/search/1657+E.+Helen+St.+%7C+Tucson,+AZ+85721+%0D%0A++++++++++++++++++++++Office:++520?entry=gmail&source=g">1657 E. Helen St. | Tucson, AZ 85721</a> <a href="https://www.google.com/maps/search/1657+E.+Helen+St.+%7C+Tucson,+AZ+85721+%0D%0A++++++++++++++++++++++Office:++520?entry=gmail&source=g"> Office: </a><a href="tel:520-626-9589" target="_blank" rel="noreferrer">520-626-9589</a> | Cell: <a href="tel:520-907-4352" target="_blank" rel="noreferrer">520-907-4352</a> <a href="mailto:+chandler@genome.arizona.edu" target="_blank" rel="noreferrer"> chandler@genome.arizona.edu </a> Pronouns: he/him/his <a href="https://www.genome.arizona.edu/" target="_blank" rel="noreferrer"> www.genome.arizona.edu </a> </td> </tr> <tr> <td> </td> <td valign="top" align="left"> <a href="https://brand.arizona.edu/signature" target="_blank" rel="noreferrer"> <img alt="Integrity, Compassion, Exploration, Adaptation, Inclusion, Determination" src="cid:part2.Hm60M6YZ.hTQZSMD7@inventati.org" width="400"> </a> </td> </tr> </tbody> </table> </div> </div> </blockquote> </div> </blockquote></div>