[systemd-devel] Systemd setup DSA interfaces in port mode and bond them together?
Brian Hutchinson
b.hutchman at gmail.com
Wed Nov 17 05:26:11 UTC 2021
Update below
On Tue, Nov 16, 2021 at 2:27 PM Brian Hutchinson <b.hutchman at gmail.com>
wrote:
> Hi Mikulėnas,
>
> On Tue, Nov 16, 2021, 3:12 AM Mantas Mikulėnas <grawity at gmail.com> wrote:
>
>> Most of this looks like it could be done with systemd-networkd to create
>> a bond .netdev, with a small oneshot service for i2c. (What's the exact
>> criteria for when it should be run? Does it depend on bond0 being there,
>> does it need to be last, etc?)
>>
>
> It can be last in the startup chain I guess, don't know what other systemd
> things that might need the network to be up before the last unit file runs.
>
> I start linuxptp too so I would have the unit file that starts ptp4l start
> after the bond was created etc.
>
> Same thing for the i2c command to enable the switch.
>
> Regards,
>
> Brian
>
>
>> On Tue, Nov 16, 2021, 02:58 Brian Hutchinson <b.hutchman at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I'm on a IMX8 platform and have a Microchip KSZ9567 Ethernet switch. I
>>> can use IP commands to manually bring lan1 and lan2 interfaces up and then
>>> create a redundant/failover bond ... but I'm having difficulty figuring out
>>> how to do this the "systemd" way.
>>>
>>> My first attempt was to just have systemd run a script of all the
>>> commands I do manually but during system startup there appears to be race
>>> conditions so I have to set my service type to "Idle" and sometimes even
>>> that doesn't work. So I want to exploit any systemd support for DSA and
>>> bonding.
>>>
>>> Here is script my manual steps which is what I want systemd to
>>> ultimately do:
>>>
>>> #!/bin/bash
>>>
>>> # Create a redundant bond between ksz9567 DSA lan1 and lan2 interfaces
>>>
>>> # Load bonding kernel module
>>> modprobe bonding
>>>
>>> # Bring up CPU interface (cpu to switch port 7 - the RGMII link)
>>> ip link set eth0 up
>>>
>>> # Create a bond
>>> echo +bond0 > /sys/class/net/bonding_masters
>>>
>>> # Set mode to active-backup (redundancy failover)
>>> echo active-backup > /sys/class/net/bond0/bonding/mode
>>>
>>> # Set time it takes (in ms) for slave to move when a link goes down
>>> echo 1000 > /sys/class/net/bond0/bonding/miimon
>>>
>>> # Add slaves to bond
>>>
>>> echo +lan1 > /sys/class/net/bond0/bonding/slaves
>>> echo +lan2 > /sys/class/net/bond0/bonding/slaves
>>>
>>> # Set IP and netmask of the bond
>>> ip addr add 192.168.0.4/24 dev bond0
>>>
>>> # And bring bond up. Pings and network connectivity should work now
>>> ip link set bond0 up
>>>
>>> # For a board that doesn't have Ethernet switch hardware strapped to
>>> enable at boot .. enable it now
>>> i2cset -f -y 0 0x5f 0x03 0x00 0x01 i
>>>
>>> Thanks for any information, pointers etc.
>>>
>>> Regards,
>>>
>>> Brian
>>>
>>
So not sure I'm doing this right. eth0 needs to be up before lan1 and lan2
can be added to the bond. Not quite sure how to do that with systemd but I
made the following files and see some progress but ping doesn't work so
appears I have no network connectivity:
root at imx8mmevk:/etc/systemd/network# cat 10-bond1.netdev
[NetDev]
Name=bond1
Kind=bond
[Bond]
Mode=active-backup
PrimaryReselectPolicy=failure
MIIMonitorSec=2s
root at imx8mmevk:/etc/systemd/network# cat 10-bond1.network
[Match]
Name=bond1
[Network]
Address=192.168.0.4/24
root at imx8mmevk:/etc/systemd/network# cat 20-lan1.network
[Match]
Name=lan1
[Network]
Bond=bond1
PrimarySlave=true
root at imx8mmevk:/etc/systemd/network# cat 30-lan2.network
[Match]
Name=lan2
[Network]
Bond=bond
ip link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode
DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1506 qdisc mq state UP mode
DEFAULT group default qlen 1000
link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
3: lan1 at eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
4: lan2 at eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode
DEFAULT group default qlen 1000
link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
5: bond1: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue
state DOWN mode DEFAULT group default qlen 1000
link/ether be:87:0a:9b:13:03 brd ff:ff:ff:ff:ff:ff
cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v5.10.69
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: None
MII Status: down
MII Polling Interval (ms): 2000
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0
At this point there should be info on lan1 and lan2 status but don't see it.
... so of course I can't ping.
Next I did systemctl restart systemd-networkd and saw the following:
systemctl restart systemd-networkd
root at imx8mmevk:~# [ 33.816313] device eth0 entered promiscuous mode
[ 33.821026] audit: type=1700 audit(1636550966.339:2): dev=eth0 prom=256
old_prom=0 auid=4294967295 uid=995 gid=994 ses=4294967295
[ 33.867164] ksz9477-switch 0-005f lan2: configuring for phy/gmii link
mode
[ 33.875066] bond1: (slave lan2): Enslaving as a backup interface with a
down link
[ 33.919055] ksz9477-switch 0-005f lan1: configuring for phy/gmii link
mode
[ 33.926683] bond1: (slave lan1): Enslaving as a backup interface with a
down link
[ 38.066148] ksz9477-switch 0-005f lan1: Link is Up - 1Gbps/Full - flow
control rx/tx
[ 39.472022] bond1: (slave lan1): link status definitely up, 1000 Mbps
full duplex
[ 39.479537] bond1: (slave lan1): making interface the new active one
[ 39.486154] bond1: active interface up!
[ 39.490071] IPv6: ADDRCONF(NETDEV_CHANGE): bond1: link becomes ready
At which point I can ping. Feels like there still might be some kind of
race condition as things won't work unless I manually issue systemctl
restart systemd-networkd command after logging in.
In kernel dmesg logs I see:
[ 4.165940] bond1: (slave lan2): Opening slave failed
[ 4.196834] bond1: (slave lan1): Opening slave failed
[ 4.315588] Generic PHY fixed-0:00: attached PHY driver [Generic PHY]
(mii_bus:phy_addr=fixed-0:00, irq=POLL)
[ 4.326181] fec 30be0000.ethernet eth0: Link is Up - 1Gbps/Full - flow
control off
[ 4.561000] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Loaded: loaded (/lib/systemd/system/systemd-networkd.service; enabled;
vendor preset: enabled)
Active: [[0;1;32mactive (running)[[0m since Sun 2020-09-20 10:43:59
UTC; 1 years 1 months ago
TriggeredBy: [[0;1;32m*[[0m systemd-networkd.socket
Docs: man:systemd-networkd.service(8)
Main PID: 251 (systemd-network)
Status: "Processing requests..."
Tasks: 1 (limit: 1574)
Memory: 1.5M
CGroup: /system.slice/systemd-networkd.service
`-251 /lib/systemd/systemd-networkd
Nov 10 13:28:56 imx8mmevk systemd-networkd[251]:
[[0;1;31m[[0;1;39m[[0;1;31mlan2:
Could not join netdev: Network is down[[0m
Nov 10 13:28:56 imx8mmevk systemd-networkd[251]:
[[0;1;38;5;185m[[0;1;39m[[0;1;38;5;185mlan2:
Failed[[0m
Nov 10 13:28:56 imx8mmevk systemd-networkd[251]:
[[0;1;31m[[0;1;39m[[0;1;31mlan1:
Could not join netdev: Network is down[[0m
Nov 10 13:28:56 imx8mmevk systemd-networkd[251]:
[[0;1;38;5;185m[[0;1;39m[[0;1;38;5;185mlan1:
Failed[[0m
Nov 10 13:28:56 imx8mmevk systemd-networkd[251]: eth0: IPv6 successfully
enabled
Nov 10 13:28:56 imx8mmevk systemd-networkd[251]: eth0: Link UP
Nov 10 13:28:57 imx8mmevk systemd-networkd[251]: eth0: Gained carrier
Nov 10 13:28:57 imx8mmevk systemd-networkd[251]: bond1: IPv6 successfully
enabled
Nov 10 13:28:57 imx8mmevk systemd-networkd[251]: bond1: Link UP
Nov 10 13:28:58 imx8mmevk systemd-networkd[251]: eth0: Gained IPv6LL
... so looks like the bond stuff is trying to happen before eth0 (my DSA
HOST/CPU interface to switch) is up. How can I make eth0 up with systemd?
eth0 just needs to be up ... no IP etc., and the bond1 gets the IP etc.
For now I'm doing the i2c command to enable my switch in u-boot but still
need to incorporate that into systemd somehow.
Any ideas as to what I'm doing wrong? I think at a minimum I need to bring
eth0 up before the bonding stuff happens but not quite sure what that would
look like using systemd.
Regards,
Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20211117/3d6f2163/attachment-0001.htm>
More information about the systemd-devel
mailing list