<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body> Hello Further to my previous email: I see that there is already an extremely similar issue reported on July 12, 2021 and it has been fixed. <a class="moz-txt-link-freetext" href="https://github.com/systemd/systemd/issues/20203">https://github.com/systemd/systemd/issues/20203</a> But I do not know if this fix exists in systemd v249.3 (Arch Linux) If it exists that means that fix is breaking my system. And if it does not exist that means, I can expect it to fix my issue. Regards, Amish. <div class="moz-cite-prefix">On 18/08/21 11:42 am, Amish wrote: </div> <blockquote type="cite" cite="mid:1043b8c7-a453-ad5e-579a-8721a7fbcfd5@gmail.com"> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> Hello, Thank you for your reply. I can understand that there can be race. But when I check logs, there is no race happening. Let us see and analyze the logs. Stage 1: System boots, and kernel assigns eth0, eth1 and eth2 as interface names. Aug 18 09:17:13 kk kernel: e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) e0:d5:5e:8d:7f:2f Aug 18 09:17:13 kk kernel: e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection Aug 18 09:17:13 kk kernel: e1000e 0000:00:1f.6 eth0: MAC: 13, PHY: 12, PBA No: FFFFFF-0FF Aug 18 09:17:13 kk kernel: 8139too 0000:04:00.0 eth1: RealTek RTL8139 at 0x000000000e8fc9bb, 00:e0:4d:05:ee:a2, IRQ 19 Aug 18 09:17:13 kk kernel: r8169 0000:02:00.0 eth2: RTL8168e/8111e, 50:3e:aa:05:2b:ca, XID 2c2, IRQ 129 Aug 18 09:17:13 kk kernel: r8169 0000:02:00.0 eth2: jumbo features [frames: 9194 bytes, tx checksumming: ko] Stage 2: Now udev rules are triggered and the interfaces are renamed to tmpeth0, tmpeth2 and tmpeth1. Aug 18 09:17:13 kk kernel: 8139too 0000:04:00.0 tmpeth2: renamed from eth1 Aug 18 09:17:13 kk kernel: e1000e 0000:00:1f.6 tmpeth0: renamed from eth0 Aug 18 09:17:13 kk kernel: r8169 0000:02:00.0 tmpeth1: renamed from eth2 Stage 3: Now my script is called and it renames interfaces to eth0, eth2 and eth1. Aug 18 09:17:13 kk kernel: e1000e 0000:00:1f.6 eth0: renamed from tmpeth0 Aug 18 09:17:14 kk kernel: r8169 0000:02:00.0 eth1: renamed from tmpeth1 Aug 18 09:17:14 kk kernel: 8139too 0000:04:00.0 eth2: renamed from tmpeth2 Effectively original interface eth1 and eth2 are swapped. While eth0 remains eth0. All these happened before systemd-networkd started and interface renaming was over by 9:17:14. Stage 4: Now systemd-networkd starts, 2 seconds after all interface have been assigned their final names. Aug 18 09:17:16 kk systemd[1]: Starting Network Configuration... Aug 18 09:17:17 kk systemd-networkd[426]: lo: Link UP Aug 18 09:17:17 kk systemd-networkd[426]: lo: Gained carrier Aug 18 09:17:17 kk systemd-networkd[426]: Enumeration completed Aug 18 09:17:17 kk systemd[1]: Started Network Configuration. Aug 18 09:17:17 kk systemd-networkd[426]: eth2: Interface name change detected, renamed to eth1. Aug 18 09:17:17 kk systemd-networkd[426]: Could not process link message: File exists Aug 18 09:17:17 kk systemd-networkd[426]: eth1: Failed Aug 18 09:17:17 kk systemd-networkd[426]: eth1: Interface name change detected, renamed to eth2. Aug 18 09:17:17 kk systemd-networkd[426]: eth1: Interface name change detected, renamed to tmpeth2. Aug 18 09:17:17 kk systemd-networkd[426]: eth0: Interface name change detected, renamed to tmpeth0. Aug 18 09:17:17 kk systemd-networkd[426]: eth2: Interface name change detected, renamed to tmpeth1. Aug 18 09:17:17 kk systemd-networkd[426]: tmpeth0: Interface name change detected, renamed to eth0. Aug 18 09:17:17 kk systemd-networkd[426]: tmpeth1: Interface name change detected, renamed to eth1. Aug 18 09:17:17 kk systemd-networkd[426]: tmpeth2: Interface name change detected, renamed to eth2. Aug 18 09:17:17 kk systemd-networkd[426]: eth1: Link UP Aug 18 09:17:17 kk systemd-networkd[426]: eth0: Link UP Aug 18 09:17:20 kk systemd-networkd[426]: eth0: Gained carrier This is when eth0 and eth1 interfaces are up and configured by systemd-networkd but eth2 is down and not configured. None of the .network configuration files match by interface names. They all match just by MAC address. # sample .network file. [Match] MACAddress=e0:d5:5e:8d:7f:2f Type=ether [Network] IgnoreCarrierLoss=yes LinkLocalAddressing=no IPv6AcceptRA=no ConfigureWithoutCarrier=true Address=192.168.25.2/24 Above error message "eth1: failed", was not showing earlier version of systemd. So recent version of systemd-networkd is doing something different and this is where something is going wrong. Stage 5: (my workaround for this issue) I wrote a new service file which restarts systemd-networkd after waiting for 10 seconds. Aug 18 09:17:27 kk systemd[1]: Stopping Network Configuration... Aug 18 09:17:27 kk systemd[1]: systemd-networkd.service: Deactivated successfully. Aug 18 09:17:27 kk systemd[1]: Stopped Network Configuration. Aug 18 09:17:27 kk systemd[1]: Starting Network Configuration... Aug 18 09:17:27 kk systemd-networkd[579]: eth1: Link UP Aug 18 09:17:27 kk systemd-networkd[579]: eth0: Link UP Aug 18 09:17:27 kk systemd-networkd[579]: eth0: Gained carrier Aug 18 09:17:27 kk systemd-networkd[579]: lo: Link UP Aug 18 09:17:27 kk systemd-networkd[579]: lo: Gained carrier Aug 18 09:17:27 kk systemd-networkd[579]: Enumeration completed Aug 18 09:17:27 kk systemd[1]: Started Network Configuration. Aug 18 09:17:27 kk systemd-networkd[579]: eth2: Link UP Aug 18 09:17:27 kk systemd-networkd[579]: eth2: Gained carrier All interfaces are now up and running as expected. Please check as I do not believe that this issue is causing any race but to me it looks like some logical change in systemd-networkd which is causing the issue. Thank you and regards, Amish <div class="moz-cite-prefix">On 17/08/21 3:18 pm, Colin Guthrie wrote: </div> <blockquote type="cite" cite="mid:5b42e102-4fc0-bc8f-4f11-fb4edcdba26b@colin.guthr.ie">Hiya, As has been said, this is racy. "Sufficiently early" is just a hope, rather than a guarantee. Perhaps something in the kernel made things more or less efficient (try booting with the old kernel to see if it helps, but as this is a race, it may only work some of the time.). Or perhaps some unit ordering changed so make this better? Perhaps udev settle units have now been dropped and thus the boot is faster and things happen in a more hotplug oriented way? Lot's of possibilities for why this no longer works (and even before it definitely wasn't a guaranteed or recommended approach). As has been said, you're best to pick a different "namespace" lan0 wan0 wan1 etc. if you can but if you can't change this due to some legacy scripts, at least pick sufficiently high ethN numbers to stay out of the way of the kernel, e.g. if you have three eth cards, then pick your names starting from e.g. 5: eth5, eth6, eth7 and thus you can avoid this dance with temporary names (although I'd still recommend using different names altogether if you can). Hope that helps. Col Amish wrote on 16/08/2021 13:38: <blockquote type="cite"> On 16/08/21 5:39 pm, Lennart Poettering wrote: <blockquote type="cite">On Mo, 16.08.21 17:31, Amish (<a class="moz-txt-link-abbreviated" href="mailto:anon.amish@gmail.com" moz-do-not-send="true">anon.amish@gmail.com</a>) wrote: <blockquote type="cite">On 16/08/21 5:25 pm, Lennart Poettering wrote: <blockquote type="cite">On Mo, 16.08.21 16:09, Amish (<a class="moz-txt-link-abbreviated" href="mailto:anon.amish@gmail.com" moz-do-not-send="true">anon.amish@gmail.com</a>) wrote: <blockquote type="cite">Some old scripts that we have expect interface names starting with eth. But those names are not predictable. So to get predictable names starting with eth*, first I temporarily rename all interface with tmpeth*. This is done via udev rules. SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="XX:XX:XX:XX:XX:XX", NAME="tmpeth0" SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="XX:XX:XX:XX:XX:YY", NAME="tmpeth1" SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="XX:XX:XX:XX:XX:ZZ", NAME="tmpeth2" Then I have a small service (script) which runs before network-pre.target to convert these names back to eth* #search for network interface with name starting from "tmpeth" and rename them to "eth" /usr/bin/find /sys/class/net -maxdepth 1 -name "tmpeth[0-9]" -type l -printf "%f\n" | while read tmpiface; do /usr/bin/ip link set dev "$tmpiface" name "$(echo $tmpiface | sed s/tmpeth/eth/)"; done This ensures that I have predictable names starting with eth*. And it is working fine from 2-3 years. Even with current issue, name assignment is working fine. </blockquote> This cannot work and is necesarily race. Stay out of the ethXYZ namespace, that's the kernel's namespace. Pick any other names, i.e. "foobar0", "foobar1", but otherwise you just have a racy racy mess, because the kernel might take the name whenever it pleases. </blockquote> No I dont think this is race. Because my script runs after Udev has finished assigning the interfaces names. </blockquote> device probing can take any time it wants. there isn't a point in time where everything is probed. </blockquote> These are internal PCI LAN cards. I believe these gets probed (and named) sufficiently early. And then we can expect names assigned by Udev to remain same. And I can see in the logs that names are not changed after my script runs. Also this has been working successfully for me from 2 or more years. But after today's update, something is breaking all the systems. Additionally just now on other system I see eth2 (instead of eth1) being renamed to eth0. I just want to know what changed and where? (Kernel or Systemd?). *Also another point is, I have set ConfigureWithoutCarrier=yes in network files and all are static IPs, so systemd-networkd should have configured the devices even if links are not up. But its not doing that anymore either after today's update.* Regards Amish. <blockquote type="cite">Lennart -- Lennart Poettering, Berlin </blockquote> </blockquote> </blockquote> </blockquote> </body> </html>