Issues regarding multiple MBIM bearer support
Aleksander Morgado
aleksander at aleksander.es
Tue Jun 15 19:26:03 UTC 2021
Hey Lech,
>
> Recently, I was investigating connectivity issues with Telit LE910-EU V2 in MBIM mode, when working in roaming. At the time, I was using ModemManager 1.16.5.
>
> In order to go further while waiting for fix on the modem side - I made a workaround, which led me to discovery of a few issues regarding multiple MBIM bearer support.
>
> Long story short:
> - At time of ModemManager 1.6.x, the modem was establishing connection properly most of the time - it was hard to reproduce the connection problem at all
> - With ModemManager 1.16.x, the modem would sometimes fail at "packet service attach" step, sometimes at "connect" step.
Can you confirm that when using MM 1.6.x you didn't see the "packet
service attach" error? That would be interesting. I don't truly recall
how many changes in the MBIM connection sequence were done between 1.6
and 1.16, but that could give us some hints to debug that further.
> - Discussion with operator and Telit yielded information, that connections failed at "connect" step, because a second PDN session was requested with the same APN
> - ModemManager 1.6.x always shown only one bearer in the list of beares through mmcli.
> - ModemManager 1.16x (which gained default EPS bearer setting - which is supposedly not supported in this modem) sometimes shown multiple bearers - always a "bearer 0" in addition to some other bearer, often greater than one.
That should totally be unrelated to any of your issues. The initial
EPS bearer status object is maintained out of the "bearer list" object
that tracks the different connection attempts with different settings.
I recall having fixed some issues more or less recently that solved a
problem like yours, but those fixes were definitely already included
in 1.16.4, so maybe there is some other issue somewhere. The fixes
were 4f7ee18d1cf1ce21bba3f2701e408ae83108b3de and
99bffc03cff060a752c0c9a4af0e7d06847c0c00, both in libmm-glib; due to
those problems comparing the bearer properties would always fail even
if they were equal, and that would have triggered creating a new
bearer with the same settings, instead of reusing the last one.
> - The cause of failures at "Packet service attach" step is still not known at this step, but issue is also reproducible using pure mbimcli commands. Issuing "detach packet service" would restore the modem to operation,
> - I tried to implement this workaround inside ModemManager (which seems to be exactly the same as [1], despite being different modem manufacturer - but with exactly same Intel chipset), but it would not work for 100% time.
That issue rings a bell; i don't think we had a proper solution out of
that CGACTT=0 hack.
> - I implemented another dirty workaround - to re-register the modem in the network, by executing 'AT+COPS=2' and 'AT+COPS=0,0" through mmcli in succession - it came out during debugging, and trying to gather as many logs from modem at the time, that this actually helps to establish the connection at the PDN level.
That would make the connection attempt quite slow, doesn't it?
> - When this is executed, but the modem is already registered in home network, this expectedly causes a dropped connection, but then MM detects this situation and reacts accordingly, quickly re-establishing the connection. This is very racey, but a similar situation could be triggered by an external factor, like the operator dropping a PDN connection.
>
MM does not automatically reconnect; you probably have NM doing that
autoreconnection for you.
> And finally, we come to the issues in ModemManager, that I discovered because of this:
> - When reconnecting after triggering the issue, ModemManager always ends up with two bearers associated with the modem, one of which is disconnected and has no IP configuration.
> MM 1.6.x did not, and this is first issue - I suspect that a bearer cleanup in 1.16.x is missing on modem de-registration forced by AT command. Fixing this would most probably fix our usecase, because we use only a single PDN connection.
> I will try with master as well, as it might be already fixed there.
Please open a bug in gitlab and provide steps to reproduce and a full
debug log of the daemon:
https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/issues/
Also, please make sure you attempt to reproduce with 1.16.6 at least
(and git master if possible)
Also, please note one thing. If you ask NetworkManager to request a
connection without specifying IP type, it will try IPv4v6 first, if
that fails it will try IPv4-only, and if that fails it will try
IPv6-pnly. If your connection attempt ends up with one bearer
disconnected and one connected, maybe the disconnected one is the
iIPv4v6 one and the connected one is the IPv4 one. Please provide the
bearer settings of both bearers once you create the gitlab issue.
> - Whether the connection actually works, depends on which bearer gets actually connected, and this is random. If it is "Bearer 0", then all is fine, IP connectivity works. If it is the other bearer, then connection does not work. The randomness here is the second issue.
> - Connectivity does not work in case of bearer different than 0, because, at MBIM level, ModemManager establishes a PDN session at ID different than "session 0", if not using "Bearer 0".
> According to Linux kernel documentation for cdc_mbim driver, such sessions are mapped as 802.1Q VLANs at the network interface level. [2]
> I confirmed this by manually creating VLAN sub-interface for the modem, with session number extracted through mbimcli, and moving the IP configuration there - this allowed the connectivity to work again.
Yep, that is a bit broken in 1.16.x and fixed in git master. In the
case of git master there is a proper fix taking into account the
multiplexing feature (see e.g.
e5305498b7ddbc5e419cf51fc0c30aa6593a40d2). That is broken in MM 1.16,
but if you had a proper bearer reconnection logic in place, it
wouldn't be needed. E.g. if there is always one single bearer being
connected in the MBIM modem, you should never hit the issue that we
attempt to use any other thing than session id 0. That was the
compromise I took for MM 1.16.
>
> Here is a CLI dump confirming my hypothesis regarding erroneous VLAN mapping:
> # mmcli -m 0
> --------------------------------
> General | path: /org/freedesktop/ModemManager1/Modem/0
> | device id: b49ab6d9407e88df9ca805de471af5ac7e066589
> --------------------------------
> Hardware | manufacturer: Telit
> | model: FIH7160
> | firmware revision: 20.00.406
> | h/w revision: XMM7160_V1.2_HWID665_MBIM_NAND
> | supported: gsm-umts, lte
> | current: gsm-umts, lte
> | equipment id: <redacted>
> --------------------------------
> System | device: /sys/devices/platform/soc/30800000.bus/30b20000.usb/ci_hdrc.1/usb1/1-1/1-1.3
> | drivers: cdc_acm, cdc_mbim
> | plugin: telit
> | primary port: cdc-wdm0
> | ports: cdc-wdm0 (mbim), ttyACM0 (at), ttyACM1 (ignored),
> | ttyACM2 (ignored), ttyACM3 (at), ttyACM4 (ignored), ttyACM5 (ignored),
> | wwan0 (net)
> --------------------------------
> Status | unlock retries: sim-pin2 (3)
> | state: connected
> | power state: on
> | access tech: lte
> | signal quality: 80% (recent)
> --------------------------------
> Modes | supported: allowed: 2g; preferred: none
> | allowed: 3g; preferred: none
> | allowed: 2g, 3g; preferred: none
> | allowed: 4g; preferred: none
> | allowed: 2g, 4g; preferred: none
> | allowed: 3g, 4g; preferred: none
> | allowed: 2g, 3g, 4g; preferred: none
> | current: allowed: 2g, 3g, 4g; preferred: none
> --------------------------------
> Bands | supported: egsm, dcs, utran-1, utran-8, eutran-1, eutran-3, eutran-7,
> | eutran-8, eutran-20
> | current: egsm, dcs, utran-1, eutran-1
> --------------------------------
> IP | supported: ipv4, ipv6, ipv4v6
> --------------------------------
> 3GPP | imei: <redacted>
> | enabled locks: fixed-dialing
> | operator id: 26006
> | operator name: Play
> | registration: home
> --------------------------------
> 3GPP EPS | ue mode of operation: csps-1
> --------------------------------
> SIM | primary sim path: /org/freedesktop/ModemManager1/SIM/0
> --------------------------------
> Bearer | paths: /org/freedesktop/ModemManager1/Bearer/1
> | /org/freedesktop/ModemManager1/Bearer/0
> # mmcli -b 0
> ----------------------------
> General | path: /org/freedesktop/ModemManager1/Bearer/0
> | type: default
> ----------------------------
> Status | connected: no
> | suspended: no
> | ip timeout: 20
> ----------------------------
> Properties | apn: internet
> | roaming: allowed
> | ip type: ipv4v6
Ah look at that, ipv4v6 is the ip type in this first bearer.
> ----------------------------
> Statistics | attempts: 2
> | attempts: 1
> | total-duration: 10
> # mmcli -b 1
> --------------------------------
> General | path: /org/freedesktop/ModemManager1/Bearer/1
> | type: default
> --------------------------------
> Status | connected: yes
> | suspended: no
> | interface: wwan0
> | ip timeout: 20
> --------------------------------
> Properties | apn: internet
> | roaming: allowed
> | ip type: ipv4
And ipv4-only in the 2nd bearer. So, this confirms my theory from
before; I'm not going to edit the email response above, this is
getting too long already... :)
If you don't want the 2 bearers, make sure you set IPv6 to disable in
the NetworkManager settings; that will probably workaround all your
issues in MM1.16.
> --------------------------------
> IPv4 configuration | method: static
> | address: 100.104.40.133
> | prefix: 24
> | gateway: 100.104.40.1
> | dns: 185.89.185.1, 89.108.202.21, 185.89.185.1
> | mtu: 1500
> --------------------------------
> Statistics | attempts: 1
> # ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 56 data bytes
> ^C
> --- 8.8.8.8 ping statistics ---
> 7 packets transmitted, 0 packets received, 100% packet loss
> # mbimcli -p -d /dev/cdc-wdm0 --query-ip-configuration=0
>
> [/dev/cdc-wdm0] IPv4 configuration available: 'none'
>
> [/dev/cdc-wdm0] IPv6 configuration available: 'none'
> # mbimcli -p -d /dev/cdc-wdm0 --query-ip-configuration=1
>
> [/dev/cdc-wdm0] IPv4 configuration available: 'address, gateway, dns, mtu'
> IP [0]: '100.104.40.133/24'
> Gateway: '100.104.40.1'
> DNS [0]: '185.89.185.1'
> DNS [1]: '89.108.202.21'
> DNS [2]: '185.89.185.1'
> DNS [3]: '89.108.202.21'
> MTU: '1500'
>
> [/dev/cdc-wdm0] IPv6 configuration available: 'mtu'
> MTU: '1500'
> # ip link add link wwan0 name wwan0.1 type vlan id 1
Wait, you shouldn't do any of this yourself. Not sure if you're
attempting to test something here.
> # ip r
> default via 100.104.40.1 dev wwan0
> 100.104.40.0/24 dev wwan0 proto kernel scope link src 100.104.40.133
> 169.254.0.0/16 dev lan2 proto kernel scope link src 169.254.120.44 metric 100
> 192.168.7.0/24 dev lan3 proto kernel scope link src 192.168.7.2
> 224.0.0.0/4 dev lan2 proto static scope link metric 100
> # ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> valid_lft forever preferred_lft forever
> inet6 ::1/128 scope host
> valid_lft forever preferred_lft forever
> 2: lan2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
> link/ether 5a:10:c0:4f:ef:c8 brd ff:ff:ff:ff:ff:ff
> inet 169.254.120.44/16 brd 169.254.255.255 scope link noprefixroute lan2
> valid_lft forever preferred_lft forever
> inet6 fe80::2ee:38b1:b49a:ce1d/64 scope link noprefixroute
> valid_lft forever preferred_lft forever
> 3: sit0 at NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
> link/sit 0.0.0.0 brd 0.0.0.0
> 4: lan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
> link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
> 5: lan3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
> link/ether c2:90:20:3e:73:dd brd ff:ff:ff:ff:ff:ff
> inet 192.168.7.2/24 brd 192.168.7.255 scope global lan3
> valid_lft forever preferred_lft forever
> inet6 fe80::c090:20ff:fe3e:73dd/64 scope link
> valid_lft forever preferred_lft forever
> 7: wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
> link/ether e2:a2:fd:7b:37:c7 brd ff:ff:ff:ff:ff:ff
> inet 100.104.40.133/24 scope global wwan0
> valid_lft forever preferred_lft forever
> 8: wwan0.1 at wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
> link/ether e2:a2:fd:7b:37:c7 brd ff:ff:ff:ff:ff:ff
> inet6 fe80::e0a2:fdff:fe7b:37c7/64 scope link
> valid_lft forever preferred_lft forever
> # ip addr del 100.104.40.133/24 dev wwan0
> # ip addr add 100.104.40.133/24 dev wwan0.1
> # ip link set dev wwan0.1 up
> # ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> valid_lft forever preferred_lft forever
> inet6 ::1/128 scope host
> valid_lft forever preferred_lft forever
> 2: lan2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
> link/ether 5a:10:c0:4f:ef:c8 brd ff:ff:ff:ff:ff:ff
> inet 169.254.120.44/16 brd 169.254.255.255 scope link noprefixroute lan2
> valid_lft forever preferred_lft forever
> inet6 fe80::2ee:38b1:b49a:ce1d/64 scope link noprefixroute
> valid_lft forever preferred_lft forever
> 3: sit0 at NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
> link/sit 0.0.0.0 brd 0.0.0.0
> 4: lan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
> link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
> 5: lan3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
> link/ether c2:90:20:3e:73:dd brd ff:ff:ff:ff:ff:ff
> inet 192.168.7.2/24 brd 192.168.7.255 scope global lan3
> valid_lft forever preferred_lft forever
> inet6 fe80::c090:20ff:fe3e:73dd/64 scope link
> valid_lft forever preferred_lft forever
> 7: wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
> link/ether e2:a2:fd:7b:37:c7 brd ff:ff:ff:ff:ff:ff
> 8: wwan0.1 at wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
> link/ether e2:a2:fd:7b:37:c7 brd ff:ff:ff:ff:ff:ff
> inet 100.104.40.133/24 scope global wwan0.1
> valid_lft forever preferred_lft forever
> inet6 fe80::e0a2:fdff:fe7b:37c7/64 scope link
> valid_lft forever preferred_lft forever
> # ip r
> 100.104.40.0/24 dev wwan0.1 proto kernel scope link src 100.104.40.133
> 169.254.0.0/16 dev lan2 proto kernel scope link src 169.254.120.44 metric 100
> 192.168.7.0/24 dev lan3 proto kernel scope link src 192.168.7.2
> 224.0.0.0/4 dev lan2 proto static scope link metric 100
> # ip r add default via 100.104.40.1
> # ip r
> default via 100.104.40.1 dev wwan0.1
> 100.104.40.0/24 dev wwan0.1 proto kernel scope link src 100.104.40.133
> 169.254.0.0/16 dev lan2 proto kernel scope link src 169.254.120.44 metric 100
> 192.168.7.0/24 dev lan3 proto kernel scope link src 192.168.7.2
> 224.0.0.0/4 dev lan2 proto static scope link metric 100
> # ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 56 data bytes
> 64 bytes from 8.8.8.8: seq=0 ttl=116 time=44.351 ms
> 64 bytes from 8.8.8.8: seq=1 ttl=116 time=34.228 ms
> 64 bytes from 8.8.8.8: seq=2 ttl=116 time=41.697 ms
> 64 bytes from 8.8.8.8: seq=3 ttl=116 time=33.472 ms
> 64 bytes from 8.8.8.8: seq=4 ttl=116 time=33.080 ms
> 64 bytes from 8.8.8.8: seq=5 ttl=116 time=32.852 ms
> ^C
> --- 8.8.8.8 ping statistics ---
> 6 packets transmitted, 6 packets received, 0% packet loss
> round-trip min/avg/max = 32.852/36.613/44.351 ms
> #
>
Ahh, ok, what you did here is setup a VLAN interface for session 1,
because it was session 1 the one active in the 2nd bearer created
above, right?
If so, yes, you did hit the bug and you found a way to confirm it's a
bug. Still, the solution for this problem is to fully avoid having 2
connected bearer objects. If you can reproduce the issue with the 2
bearers and the 2nd one getting session id 1, when using MM 1.16.6,
again, please provide a MM daemon debug log and post it to gitlab for
analysis.
> ModemManager only recently gained full support for PDN multiplexing. For full support of this, ModemManager most probably should pass the information about used PDN session index to NetworkManager,
> which would then setup the VLAN sub-interface accordingly, which NetworkManager, as of 1.26.8 currently does not - IP configuration is always applied to main interface.
> Therefore, stable versions probably should avoid using PDN sessions other than zero.
>
Stable 1.16 should indeed always use PDN session id 0.
In git master, the VLAN interface is created by ModemManager itself,
and we report to NetworkManager the name of the interface to be used.
All that works transparently for the user. If you switch to MM git
master, all that should work correctly without additional changes in
your side anywhere.
> Now after this very long and convoluted story, my question is: Is there any particular fix on master branch, that could work for me, or is it possible to disable multiplexing explicitly? I think I could alter what's done in commit 05b005951710c8810538ec448e89e32f28c3d592 [3] to disable multiplexing entirely - as I don't need it currently. Cherry-picking this commit verbatim would not work, as cdc_mbim driver does support multiplexing, and currently, check is done only against mhi_net driver. Still, a fix regarding bearer cleanup would be much more appreciated, or a hint on where to start looking, so I could develop and upstream the fix.
Quick workaround: disable IPv6 in the NM settings.
Final solution: use MM git master as it is. Even if you don't need
multiplexing, using it is straightforward, and all that should be
correctly working (with any version of NetworkManager).
> Thanks in advance for any information.
>
> [1] https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/issues/188
> [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/cdc_mbim.rst?id=92f06f4226fd9bdd6fbbd2e8b84601fc14b5855e#n173
> [3] https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/commit/05b005951710c8810538ec448e89e32f28c3d592
>
That was a long email; hope any of the things I said were helpful for you :)
--
Aleksander
https://aleksander.es
More information about the ModemManager-devel
mailing list