Issues regarding multiple MBIM bearer support
Lech Perczak
l.perczak at camlintechnologies.com
Tue Jun 15 13:27:05 UTC 2021
Hello,
Recently, I was investigating connectivity issues with Telit LE910-EU V2 in MBIM mode, when working in roaming. At the time, I was using ModemManager 1.16.5.
In order to go further while waiting for fix on the modem side - I made a workaround, which led me to discovery of a few issues regarding multiple MBIM bearer support.
Long story short:
- At time of ModemManager 1.6.x, the modem was establishing connection properly most of the time - it was hard to reproduce the connection problem at all
- With ModemManager 1.16.x, the modem would sometimes fail at "packet service attach" step, sometimes at "connect" step.
- Discussion with operator and Telit yielded information, that connections failed at "connect" step, because a second PDN session was requested with the same APN
- ModemManager 1.6.x always shown only one bearer in the list of beares through mmcli.
- ModemManager 1.16x (which gained default EPS bearer setting - which is supposedly not supported in this modem) sometimes shown multiple bearers - always a "bearer 0" in addition to some other bearer, often greater than one.
- The cause of failures at "Packet service attach" step is still not known at this step, but issue is also reproducible using pure mbimcli commands. Issuing "detach packet service" would restore the modem to operation,
- I tried to implement this workaround inside ModemManager (which seems to be exactly the same as [1], despite being different modem manufacturer - but with exactly same Intel chipset), but it would not work for 100% time.
- I implemented another dirty workaround - to re-register the modem in the network, by executing 'AT+COPS=2' and 'AT+COPS=0,0" through mmcli in succession - it came out during debugging, and trying to gather as many logs from modem at the time, that this actually helps to establish the connection at the PDN level.
- When this is executed, but the modem is already registered in home network, this expectedly causes a dropped connection, but then MM detects this situation and reacts accordingly, quickly re-establishing the connection. This is very racey, but a similar situation could be triggered by an external factor, like the operator dropping a PDN connection.
And finally, we come to the issues in ModemManager, that I discovered because of this:
- When reconnecting after triggering the issue, ModemManager always ends up with two bearers associated with the modem, one of which is disconnected and has no IP configuration.
MM 1.6.x did not, and this is first issue - I suspect that a bearer cleanup in 1.16.x is missing on modem de-registration forced by AT command. Fixing this would most probably fix our usecase, because we use only a single PDN connection.
I will try with master as well, as it might be already fixed there.
- Whether the connection actually works, depends on which bearer gets actually connected, and this is random. If it is "Bearer 0", then all is fine, IP connectivity works. If it is the other bearer, then connection does not work. The randomness here is the second issue.
- Connectivity does not work in case of bearer different than 0, because, at MBIM level, ModemManager establishes a PDN session at ID different than "session 0", if not using "Bearer 0".
According to Linux kernel documentation for cdc_mbim driver, such sessions are mapped as 802.1Q VLANs at the network interface level. [2]
I confirmed this by manually creating VLAN sub-interface for the modem, with session number extracted through mbimcli, and moving the IP configuration there - this allowed the connectivity to work again.
Here is a CLI dump confirming my hypothesis regarding erroneous VLAN mapping:
# mmcli -m 0
--------------------------------
General | path: /org/freedesktop/ModemManager1/Modem/0
| device id: b49ab6d9407e88df9ca805de471af5ac7e066589
--------------------------------
Hardware | manufacturer: Telit
| model: FIH7160
| firmware revision: 20.00.406
| h/w revision: XMM7160_V1.2_HWID665_MBIM_NAND
| supported: gsm-umts, lte
| current: gsm-umts, lte
| equipment id: <redacted>
--------------------------------
System | device: /sys/devices/platform/soc/30800000.bus/30b20000.usb/ci_hdrc.1/usb1/1-1/1-1.3
| drivers: cdc_acm, cdc_mbim
| plugin: telit
| primary port: cdc-wdm0
| ports: cdc-wdm0 (mbim), ttyACM0 (at), ttyACM1 (ignored),
| ttyACM2 (ignored), ttyACM3 (at), ttyACM4 (ignored), ttyACM5 (ignored),
| wwan0 (net)
--------------------------------
Status | unlock retries: sim-pin2 (3)
| state: connected
| power state: on
| access tech: lte
| signal quality: 80% (recent)
--------------------------------
Modes | supported: allowed: 2g; preferred: none
| allowed: 3g; preferred: none
| allowed: 2g, 3g; preferred: none
| allowed: 4g; preferred: none
| allowed: 2g, 4g; preferred: none
| allowed: 3g, 4g; preferred: none
| allowed: 2g, 3g, 4g; preferred: none
| current: allowed: 2g, 3g, 4g; preferred: none
--------------------------------
Bands | supported: egsm, dcs, utran-1, utran-8, eutran-1, eutran-3, eutran-7,
| eutran-8, eutran-20
| current: egsm, dcs, utran-1, eutran-1
--------------------------------
IP | supported: ipv4, ipv6, ipv4v6
--------------------------------
3GPP | imei: <redacted>
| enabled locks: fixed-dialing
| operator id: 26006
| operator name: Play
| registration: home
--------------------------------
3GPP EPS | ue mode of operation: csps-1
--------------------------------
SIM | primary sim path: /org/freedesktop/ModemManager1/SIM/0
--------------------------------
Bearer | paths: /org/freedesktop/ModemManager1/Bearer/1
| /org/freedesktop/ModemManager1/Bearer/0
# mmcli -b 0
----------------------------
General | path: /org/freedesktop/ModemManager1/Bearer/0
| type: default
----------------------------
Status | connected: no
| suspended: no
| ip timeout: 20
----------------------------
Properties | apn: internet
| roaming: allowed
| ip type: ipv4v6
----------------------------
Statistics | attempts: 2
| attempts: 1
| total-duration: 10
# mmcli -b 1
--------------------------------
General | path: /org/freedesktop/ModemManager1/Bearer/1
| type: default
--------------------------------
Status | connected: yes
| suspended: no
| interface: wwan0
| ip timeout: 20
--------------------------------
Properties | apn: internet
| roaming: allowed
| ip type: ipv4
--------------------------------
IPv4 configuration | method: static
| address: 100.104.40.133
| prefix: 24
| gateway: 100.104.40.1
| dns: 185.89.185.1, 89.108.202.21, 185.89.185.1
| mtu: 1500
--------------------------------
Statistics | attempts: 1
# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
7 packets transmitted, 0 packets received, 100% packet loss
# mbimcli -p -d /dev/cdc-wdm0 --query-ip-configuration=0
[/dev/cdc-wdm0] IPv4 configuration available: 'none'
[/dev/cdc-wdm0] IPv6 configuration available: 'none'
# mbimcli -p -d /dev/cdc-wdm0 --query-ip-configuration=1
[/dev/cdc-wdm0] IPv4 configuration available: 'address, gateway, dns, mtu'
IP [0]: '100.104.40.133/24'
Gateway: '100.104.40.1'
DNS [0]: '185.89.185.1'
DNS [1]: '89.108.202.21'
DNS [2]: '185.89.185.1'
DNS [3]: '89.108.202.21'
MTU: '1500'
[/dev/cdc-wdm0] IPv6 configuration available: 'mtu'
MTU: '1500'
# ip link add link wwan0 name wwan0.1 type vlan id 1
# ip r
default via 100.104.40.1 dev wwan0
100.104.40.0/24 dev wwan0 proto kernel scope link src 100.104.40.133
169.254.0.0/16 dev lan2 proto kernel scope link src 169.254.120.44 metric 100
192.168.7.0/24 dev lan3 proto kernel scope link src 192.168.7.2
224.0.0.0/4 dev lan2 proto static scope link metric 100
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: lan2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 5a:10:c0:4f:ef:c8 brd ff:ff:ff:ff:ff:ff
inet 169.254.120.44/16 brd 169.254.255.255 scope link noprefixroute lan2
valid_lft forever preferred_lft forever
inet6 fe80::2ee:38b1:b49a:ce1d/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: sit0 at NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/sit 0.0.0.0 brd 0.0.0.0
4: lan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
5: lan3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether c2:90:20:3e:73:dd brd ff:ff:ff:ff:ff:ff
inet 192.168.7.2/24 brd 192.168.7.255 scope global lan3
valid_lft forever preferred_lft forever
inet6 fe80::c090:20ff:fe3e:73dd/64 scope link
valid_lft forever preferred_lft forever
7: wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether e2:a2:fd:7b:37:c7 brd ff:ff:ff:ff:ff:ff
inet 100.104.40.133/24 scope global wwan0
valid_lft forever preferred_lft forever
8: wwan0.1 at wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether e2:a2:fd:7b:37:c7 brd ff:ff:ff:ff:ff:ff
inet6 fe80::e0a2:fdff:fe7b:37c7/64 scope link
valid_lft forever preferred_lft forever
# ip addr del 100.104.40.133/24 dev wwan0
# ip addr add 100.104.40.133/24 dev wwan0.1
# ip link set dev wwan0.1 up
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: lan2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 5a:10:c0:4f:ef:c8 brd ff:ff:ff:ff:ff:ff
inet 169.254.120.44/16 brd 169.254.255.255 scope link noprefixroute lan2
valid_lft forever preferred_lft forever
inet6 fe80::2ee:38b1:b49a:ce1d/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: sit0 at NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/sit 0.0.0.0 brd 0.0.0.0
4: lan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
5: lan3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether c2:90:20:3e:73:dd brd ff:ff:ff:ff:ff:ff
inet 192.168.7.2/24 brd 192.168.7.255 scope global lan3
valid_lft forever preferred_lft forever
inet6 fe80::c090:20ff:fe3e:73dd/64 scope link
valid_lft forever preferred_lft forever
7: wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether e2:a2:fd:7b:37:c7 brd ff:ff:ff:ff:ff:ff
8: wwan0.1 at wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether e2:a2:fd:7b:37:c7 brd ff:ff:ff:ff:ff:ff
inet 100.104.40.133/24 scope global wwan0.1
valid_lft forever preferred_lft forever
inet6 fe80::e0a2:fdff:fe7b:37c7/64 scope link
valid_lft forever preferred_lft forever
# ip r
100.104.40.0/24 dev wwan0.1 proto kernel scope link src 100.104.40.133
169.254.0.0/16 dev lan2 proto kernel scope link src 169.254.120.44 metric 100
192.168.7.0/24 dev lan3 proto kernel scope link src 192.168.7.2
224.0.0.0/4 dev lan2 proto static scope link metric 100
# ip r add default via 100.104.40.1
# ip r
default via 100.104.40.1 dev wwan0.1
100.104.40.0/24 dev wwan0.1 proto kernel scope link src 100.104.40.133
169.254.0.0/16 dev lan2 proto kernel scope link src 169.254.120.44 metric 100
192.168.7.0/24 dev lan3 proto kernel scope link src 192.168.7.2
224.0.0.0/4 dev lan2 proto static scope link metric 100
# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=116 time=44.351 ms
64 bytes from 8.8.8.8: seq=1 ttl=116 time=34.228 ms
64 bytes from 8.8.8.8: seq=2 ttl=116 time=41.697 ms
64 bytes from 8.8.8.8: seq=3 ttl=116 time=33.472 ms
64 bytes from 8.8.8.8: seq=4 ttl=116 time=33.080 ms
64 bytes from 8.8.8.8: seq=5 ttl=116 time=32.852 ms
^C
--- 8.8.8.8 ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 32.852/36.613/44.351 ms
#
ModemManager only recently gained full support for PDN multiplexing. For full support of this, ModemManager most probably should pass the information about used PDN session index to NetworkManager,
which would then setup the VLAN sub-interface accordingly, which NetworkManager, as of 1.26.8 currently does not - IP configuration is always applied to main interface.
Therefore, stable versions probably should avoid using PDN sessions other than zero.
Now after this very long and convoluted story, my question is: Is there any particular fix on master branch, that could work for me, or is it possible to disable multiplexing explicitly? I think I could alter what's done in commit 05b005951710c8810538ec448e89e32f28c3d592 [3] to disable multiplexing entirely - as I don't need it currently. Cherry-picking this commit verbatim would not work, as cdc_mbim driver does support multiplexing, and currently, check is done only against mhi_net driver. Still, a fix regarding bearer cleanup would be much more appreciated, or a hint on where to start looking, so I could develop and upstream the fix.
Thanks in advance for any information.
[1] https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/issues/188
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/cdc_mbim.rst?id=92f06f4226fd9bdd6fbbd2e8b84601fc14b5855e#n173
[3] https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/commit/05b005951710c8810538ec448e89e32f28c3d592
--
Pozdrawiam/With kind regards,
Lech Perczak
Sr. Software Engineer
Camlin Technologies Poland Limited Sp. z o.o.
More information about the ModemManager-devel
mailing list