Issues regarding multiple MBIM bearer support

Mon Jun 21 10:02:00 UTC 2021

Hi Aleksander,

Just to let you know - I cherry-picked e5305498b7ddbc5e419cf51fc0c30aa6593a40d2 over 1.16.6 tag, and it did fix the VLAN issue - new bearers objects appear, but all use session 0.
Without the fix it didn't - the issue was caught exactly on vanilla 1.16.6, but we didn't gather the logs at the time.

I prefer this over updating configuration on units deployed in the field. I can still provide you with the logs if you wish.

With master (83ac82470589a3672092a0ba0be855093b1cf5e2) I had problems. MM did want to enable multiplexing, but it failed for some reason, probably at sub-interface creation:

2021-06-16 10:04:05.630664 +00:00 test-host NetworkManager[457]: <INFO>  <info>  [1623837845.6302] modem["cdc-wdm0"]: modem state changed, 'registered' --> 'connecting' (reason: user-r
equested)
2021-06-16 10:04:05.636959 +00:00 test-host ModemManager[455]: <DEBUG>  Using dynamic session ID 0
2021-06-16 10:04:05.637483 +00:00 test-host ModemManager[455]: <WARNING>  <warn>  [modem0/bearer4] connection attempt #1 failed: failed to create net link for device: failed to add lin
k for device: Could not allocate link: Failed to add link with session id 0: Netlink message with transaction 5 failed: Unknown error -1
2021-06-16 10:04:05.638678 +00:00 test-host ModemManager[455]: <INFO>  <info>  [modem0] state changed (connecting -> registered) 

On the device I have 5.11.19 kernel with 802.1Q enabled. Again, I can provide you with detailed logs if this wasn't fixed already.

Kind regards,
Lech

W dniu 15.06.2021 o 21:26, Aleksander Morgado pisze:
> Hey Lech,
>
> >
> > Recently, I was investigating connectivity issues with Telit LE910-EU V2 in MBIM mode, when working in roaming. At the time, I was using ModemManager 1.16.5.
> >
> > In order to go further while waiting for fix on the modem side - I made a workaround, which led me to discovery of a few issues regarding multiple MBIM bearer support.
> >
> > Long story short:
> > - At time of ModemManager 1.6.x, the modem was establishing connection properly most of the time - it was hard to reproduce the connection problem at all
> > - With ModemManager 1.16.x, the modem would sometimes fail at "packet service attach" step, sometimes at "connect" step.
>
> Can you confirm that when using MM 1.6.x you didn't see the "packet
> service attach" error? That would be interesting. I don't truly recall
> how many changes in the MBIM connection sequence were done between 1.6
> and 1.16, but that could give us some hints to debug that further.
>
> > - Discussion with operator and Telit yielded information, that connections failed at "connect" step, because a second PDN session was requested with the same APN
> > - ModemManager 1.6.x always shown only one bearer in the list of beares through mmcli.
> > - ModemManager 1.16x (which gained default EPS bearer setting - which is supposedly not supported in this modem) sometimes shown multiple bearers - always a "bearer 0" in addition to some other bearer, often greater than one.
>
> That should totally be unrelated to any of your issues. The initial
> EPS bearer status object is maintained out of the "bearer list" object
> that tracks the different connection attempts with different settings.
>
> I recall having fixed some issues more or less recently that solved a
> problem like yours, but those fixes were definitely already included
> in 1.16.4, so maybe there is some other issue somewhere. The fixes
> were 4f7ee18d1cf1ce21bba3f2701e408ae83108b3de and
> 99bffc03cff060a752c0c9a4af0e7d06847c0c00, both in libmm-glib; due to
> those problems comparing the bearer properties would always fail even
> if they were equal, and that would have triggered creating a new
> bearer with the same settings, instead of reusing the last one.
>
> > - The cause of failures at "Packet service attach" step is still not known at this step, but issue is also reproducible using pure mbimcli commands. Issuing "detach packet service" would restore the modem to operation,
> > - I tried to implement this workaround inside ModemManager (which seems to be exactly the same as [1], despite being different modem manufacturer - but with exactly same Intel chipset), but it would not work for 100% time.
>
> That issue rings a bell; i don't think we had a proper solution out of
> that CGACTT=0 hack.
>
> > - I implemented another dirty workaround - to re-register the modem in the network, by executing 'AT+COPS=2' and 'AT+COPS=0,0" through mmcli in succession - it came out during debugging, and trying to gather as many logs from modem at the time, that this actually helps to establish the connection at the PDN level.
>
> That would make the connection attempt quite slow, doesn't it?
>
> > - When this is executed, but the modem is already registered in home network, this expectedly causes a dropped connection, but then MM detects this situation and reacts accordingly, quickly re-establishing the connection. This is very racey, but a similar situation could be triggered by an external factor, like the operator dropping a PDN connection.
> >
>
> MM does not automatically reconnect; you probably have NM doing that
> autoreconnection for you.
>
> > And finally, we come to the issues in ModemManager, that I discovered because of this:
> > - When reconnecting after triggering the issue, ModemManager always ends up with two bearers associated with the modem, one of which is disconnected and has no IP configuration.
> > MM 1.6.x did not, and this is first issue - I suspect that a bearer cleanup in 1.16.x is missing on modem de-registration forced by AT command. Fixing this would most probably fix our usecase, because we use only a single PDN connection.
> > I will try with master as well, as it might be already fixed there.
>
> Please open a bug in gitlab and provide steps to reproduce and a full
> debug log of the daemon:
> https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/issues/ <https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/issues>
> Also, please make sure you attempt to reproduce with 1.16.6 at least
> (and git master if possible)
>
> Also, please note one thing. If you ask NetworkManager to request a
> connection without specifying IP type, it will try IPv4v6 first, if
> that fails it will try IPv4-only, and if that fails it will try
> IPv6-pnly. If your connection attempt ends up with one bearer
> disconnected and one connected, maybe the disconnected one is the
> iIPv4v6 one and the connected one is the IPv4 one. Please provide the
> bearer settings of both bearers once you create the gitlab issue.
>
> > - Whether the connection actually works, depends on which bearer gets actually connected, and this is random. If it is "Bearer 0", then all is fine, IP connectivity works. If it is the other bearer, then connection does not work. The randomness here is the second issue.
> > - Connectivity does not work in case of bearer different than 0, because, at MBIM level, ModemManager establishes a PDN session at ID different than "session 0", if not using "Bearer 0".
> > According to Linux kernel documentation for cdc_mbim driver, such sessions are mapped as 802.1Q VLANs at the network interface level. [2]
> > I confirmed this by manually creating VLAN sub-interface for the modem, with session number extracted through mbimcli, and moving the IP configuration there - this allowed the connectivity to work again.
>
> Yep, that is a bit broken in 1.16.x and fixed in git master. In the
> case of git master there is a proper fix taking into account the
> multiplexing feature (see e.g.
> e5305498b7ddbc5e419cf51fc0c30aa6593a40d2). That is broken in MM 1.16,
> but if you had a proper bearer reconnection logic in place, it
> wouldn't be needed. E.g. if there is always one single bearer being
> connected in the MBIM modem, you should never hit the issue that we
> attempt to use any other thing than session id 0. That was the
> compromise I took for MM 1.16.
>
> >
> > Here is a CLI dump confirming my hypothesis regarding erroneous VLAN mapping:
> > # mmcli -m 0
> > --------------------------------
> > General | path: /org/freedesktop/ModemManager1/Modem/0
> > | device id: b49ab6d9407e88df9ca805de471af5ac7e066589
> > --------------------------------
> > Hardware | manufacturer: Telit
> > | model: FIH7160
> > | firmware revision: 20.00.406
> > | h/w revision: XMM7160_V1.2_HWID665_MBIM_NAND
> > | supported: gsm-umts, lte
> > | current: gsm-umts, lte
> > | equipment id: <redacted>
> > --------------------------------
> > System | device: /sys/devices/platform/soc/30800000.bus/30b20000.usb/ci_hdrc.1/usb1/1-1/1-1.3
> > | drivers: cdc_acm, cdc_mbim
> > | plugin: telit
> > | primary port: cdc-wdm0
> > | ports: cdc-wdm0 (mbim), ttyACM0 (at), ttyACM1 (ignored),
> > | ttyACM2 (ignored), ttyACM3 (at), ttyACM4 (ignored), ttyACM5 (ignored),
> > | wwan0 (net)
> > --------------------------------
> > Status | unlock retries: sim-pin2 (3)
> > | state: connected
> > | power state: on
> > | access tech: lte
> > | signal quality: 80% (recent)
> > --------------------------------
> > Modes | supported: allowed: 2g; preferred: none
> > | allowed: 3g; preferred: none
> > | allowed: 2g, 3g; preferred: none
> > | allowed: 4g; preferred: none
> > | allowed: 2g, 4g; preferred: none
> > | allowed: 3g, 4g; preferred: none
> > | allowed: 2g, 3g, 4g; preferred: none
> > | current: allowed: 2g, 3g, 4g; preferred: none
> > --------------------------------
> > Bands | supported: egsm, dcs, utran-1, utran-8, eutran-1, eutran-3, eutran-7,
> > | eutran-8, eutran-20
> > | current: egsm, dcs, utran-1, eutran-1
> > --------------------------------
> > IP | supported: ipv4, ipv6, ipv4v6
> > --------------------------------
> > 3GPP | imei: <redacted>
> > | enabled locks: fixed-dialing
> > | operator id: 26006
> > | operator name: Play
> > | registration: home
> > --------------------------------
> > 3GPP EPS | ue mode of operation: csps-1
> > --------------------------------
> > SIM | primary sim path: /org/freedesktop/ModemManager1/SIM/0
> > --------------------------------
> > Bearer | paths: /org/freedesktop/ModemManager1/Bearer/1
> > | /org/freedesktop/ModemManager1/Bearer/0
> > # mmcli -b 0
> > ----------------------------
> > General | path: /org/freedesktop/ModemManager1/Bearer/0
> > | type: default
> > ----------------------------
> > Status | connected: no
> > | suspended: no
> > | ip timeout: 20
> > ----------------------------
> > Properties | apn: internet
> > | roaming: allowed
> > | ip type: ipv4v6
>
> Ah look at that, ipv4v6 is the ip type in this first bearer.
>
> > ----------------------------
> > Statistics | attempts: 2
> > | attempts: 1
> > | total-duration: 10
> > # mmcli -b 1
> > --------------------------------
> > General | path: /org/freedesktop/ModemManager1/Bearer/1
> > | type: default
> > --------------------------------
> > Status | connected: yes
> > | suspended: no
> > | interface: wwan0
> > | ip timeout: 20
> > --------------------------------
> > Properties | apn: internet
> > | roaming: allowed
> > | ip type: ipv4
>
> And ipv4-only in the 2nd bearer. So, this confirms my theory from
> before; I'm not going to edit the email response above, this is
> getting too long already... :)
>
> If you don't want the 2 bearers, make sure you set IPv6 to disable in
> the NetworkManager settings; that will probably workaround all your
> issues in MM1.16.
>
> > --------------------------------
> > IPv4 configuration | method: static
> > | address: 100.104.40.133 <http://100.104.40.133>
> > | prefix: 24
> > | gateway: 100.104.40.1 <http://100.104.40.1>
> > | dns: 185.89.185.1 <http://185.89.185.1>, 89.108.202.21 <http://89.108.202.21>, 185.89.185.1 <http://185.89.185.1>
> > | mtu: 1500
> > --------------------------------
> > Statistics | attempts: 1
> > # ping 8.8.8.8 <http://8.8.8.8>
> > PING 8.8.8.8 <http://8.8.8.8> (8.8.8.8 <http://8.8.8.8>: 56 data bytes
> > ^C
> > --- 8.8.8.8 <http://8.8.8.8> ping statistics ---
> > 7 packets transmitted, 0 packets received, 100% packet loss
> > # mbimcli -p -d /dev/cdc-wdm0 --query-ip-configuration=0
> >
> > [/dev/cdc-wdm0] IPv4 configuration available: 'none'
> >
> > [/dev/cdc-wdm0] IPv6 configuration available: 'none'
> > # mbimcli -p -d /dev/cdc-wdm0 --query-ip-configuration=1
> >
> > [/dev/cdc-wdm0] IPv4 configuration available: 'address, gateway, dns, mtu'
> > IP [0]: '100.104.40.133/24 <http://100.104.40.133/24>'
> > Gateway: '100.104.40.1 <http://100.104.40.1>'
> > DNS [0]: '185.89.185.1 <http://185.89.185.1>'
> > DNS [1]: '89.108.202.21 <http://89.108.202.21>'
> > DNS [2]: '185.89.185.1 <http://185.89.185.1>'
> > DNS [3]: '89.108.202.21 <http://89.108.202.21>'
> > MTU: '1500'
> >
> > [/dev/cdc-wdm0] IPv6 configuration available: 'mtu'
> > MTU: '1500'
> > # ip link add link wwan0 name wwan0.1 type vlan id 1
>
> Wait, you shouldn't do any of this yourself. Not sure if you're
> attempting to test something here.
>
> > # ip r
> > default via 100.104.40.1 <http://100.104.40.1> dev wwan0
> > 100.104.40.0/24 <http://100.104.40.0/24> dev wwan0 proto kernel scope link src 100.104.40.133 <http://100.104.40.133>
> > 169.254.0.0/16 <http://169.254.0.0/16> dev lan2 proto kernel scope link src 169.254.120.44 <http://169.254.120.44> metric 100
> > 192.168.7.0/24 <http://192.168.7.0/24> dev lan3 proto kernel scope link src 192.168.7.2
> > 224.0.0.0/4 <http://224.0.0.0/4> dev lan2 proto static scope link metric 100
> > # ip a
> > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
> > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> > inet 127.0.0.1/8 <http://127.0.0.1/8> scope host lo
> > valid_lft forever preferred_lft forever
> > inet6 ::1/128 scope host
> > valid_lft forever preferred_lft forever
> > 2: lan2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
> > link/ether 5a:10:c0:4f:ef:c8 brd ff:ff:ff:ff:ff:ff
> > inet 169.254.120.44/16 <http://169.254.120.44/16> brd 169.254.255.255 <http://169.254.255.255> scope link noprefixroute lan2
> > valid_lft forever preferred_lft forever
> > inet6 fe80::2ee:38b1:b49a:ce1d/64 scope link noprefixroute
> > valid_lft forever preferred_lft forever
> > 3: sit0 at NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
> > link/sit 0.0.0.0 brd 0.0.0.0
> > 4: lan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
> > link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
> > 5: lan3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
> > link/ether c2:90:20:3e:73:dd brd ff:ff:ff:ff:ff:ff
> > inet 192.168.7.2/24 <http://192.168.7.2/24> brd 192.168.7.255 scope global lan3
> > valid_lft forever preferred_lft forever
> > inet6 fe80::c090:20ff:fe3e:73dd/64 scope link
> > valid_lft forever preferred_lft forever
> > 7: wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
> > link/ether e2:a2:fd:7b:37:c7 brd ff:ff:ff:ff:ff:ff
> > inet 100.104.40.133/24 <http://100.104.40.133/24> scope global wwan0
> > valid_lft forever preferred_lft forever
> > 8: wwan0.1 at wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
> > link/ether e2:a2:fd:7b:37:c7 brd ff:ff:ff:ff:ff:ff
> > inet6 fe80::e0a2:fdff:fe7b:37c7/64 scope link
> > valid_lft forever preferred_lft forever
> > # ip addr del 100.104.40.133/24 <http://100.104.40.133/24> dev wwan0
> > # ip addr add 100.104.40.133/24 <http://100.104.40.133/24> dev wwan0.1
> > # ip link set dev wwan0.1 up
> > # ip a
> > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
> > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> > inet 127.0.0.1/8 <http://127.0.0.1/8> scope host lo
> > valid_lft forever preferred_lft forever
> > inet6 ::1/128 scope host
> > valid_lft forever preferred_lft forever
> > 2: lan2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
> > link/ether 5a:10:c0:4f:ef:c8 brd ff:ff:ff:ff:ff:ff
> > inet 169.254.120.44/16 <http://169.254.120.44/16> brd 169.254.255.255 <http://169.254.255.255> scope link noprefixroute lan2
> > valid_lft forever preferred_lft forever
> > inet6 fe80::2ee:38b1:b49a:ce1d/64 scope link noprefixroute
> > valid_lft forever preferred_lft forever
> > 3: sit0 at NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
> > link/sit 0.0.0.0 brd 0.0.0.0
> > 4: lan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
> > link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
> > 5: lan3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
> > link/ether c2:90:20:3e:73:dd brd ff:ff:ff:ff:ff:ff
> > inet 192.168.7.2/24 <http://192.168.7.2/24> brd 192.168.7.255 scope global lan3
> > valid_lft forever preferred_lft forever
> > inet6 fe80::c090:20ff:fe3e:73dd/64 scope link
> > valid_lft forever preferred_lft forever
> > 7: wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
> > link/ether e2:a2:fd:7b:37:c7 brd ff:ff:ff:ff:ff:ff
> > 8: wwan0.1 at wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
> > link/ether e2:a2:fd:7b:37:c7 brd ff:ff:ff:ff:ff:ff
> > inet 100.104.40.133/24 <http://100.104.40.133/24> scope global wwan0.1
> > valid_lft forever preferred_lft forever
> > inet6 fe80::e0a2:fdff:fe7b:37c7/64 scope link
> > valid_lft forever preferred_lft forever
> > # ip r
> > 100.104.40.0/24 <http://100.104.40.0/24> dev wwan0.1 proto kernel scope link src 100.104.40.133 <http://100.104.40.133>
> > 169.254.0.0/16 <http://169.254.0.0/16> dev lan2 proto kernel scope link src 169.254.120.44 <http://169.254.120.44> metric 100
> > 192.168.7.0/24 <http://192.168.7.0/24> dev lan3 proto kernel scope link src 192.168.7.2
> > 224.0.0.0/4 <http://224.0.0.0/4> dev lan2 proto static scope link metric 100
> > # ip r add default via 100.104.40.1 <http://100.104.40.1>
> > # ip r
> > default via 100.104.40.1 <http://100.104.40.1> dev wwan0.1
> > 100.104.40.0/24 <http://100.104.40.0/24> dev wwan0.1 proto kernel scope link src 100.104.40.133 <http://100.104.40.133>
> > 169.254.0.0/16 <http://169.254.0.0/16> dev lan2 proto kernel scope link src 169.254.120.44 <http://169.254.120.44> metric 100
> > 192.168.7.0/24 <http://192.168.7.0/24> dev lan3 proto kernel scope link src 192.168.7.2
> > 224.0.0.0/4 <http://224.0.0.0/4> dev lan2 proto static scope link metric 100
> > # ping 8.8.8.8 <http://8.8.8.8>
> > PING 8.8.8.8 <http://8.8.8.8> (8.8.8.8 <http://8.8.8.8>: 56 data bytes
> > 64 bytes from 8.8.8.8 <http://8.8.8.8> seq=0 ttl=116 time=44.351 ms
> > 64 bytes from 8.8.8.8 <http://8.8.8.8> seq=1 ttl=116 time=34.228 ms
> > 64 bytes from 8.8.8.8 <http://8.8.8.8> seq=2 ttl=116 time=41.697 ms
> > 64 bytes from 8.8.8.8 <http://8.8.8.8> seq=3 ttl=116 time=33.472 ms
> > 64 bytes from 8.8.8.8 <http://8.8.8.8> seq=4 ttl=116 time=33.080 ms
> > 64 bytes from 8.8.8.8 <http://8.8.8.8> seq=5 ttl=116 time=32.852 ms
> > ^C
> > --- 8.8.8.8 <http://8.8.8.8> ping statistics ---
> > 6 packets transmitted, 6 packets received, 0% packet loss
> > round-trip min/avg/max = 32.852/36.613/44.351 ms
> > #
> >
>
> Ahh, ok, what you did here is setup a VLAN interface for session 1,
> because it was session 1 the one active in the 2nd bearer created
> above, right?
> If so, yes, you did hit the bug and you found a way to confirm it's a
> bug. Still, the solution for this problem is to fully avoid having 2
> connected bearer objects. If you can reproduce the issue with the 2
> bearers and the 2nd one getting session id 1, when using MM 1.16.6,
> again, please provide a MM daemon debug log and post it to gitlab for
> analysis.
>
>
> > ModemManager only recently gained full support for PDN multiplexing. For full support of this, ModemManager most probably should pass the information about used PDN session index to NetworkManager,
> > which would then setup the VLAN sub-interface accordingly, which NetworkManager, as of 1.26.8 currently does not - IP configuration is always applied to main interface.
> > Therefore, stable versions probably should avoid using PDN sessions other than zero.
> >
>
> Stable 1.16 should indeed always use PDN session id 0.
>
> In git master, the VLAN interface is created by ModemManager itself,
> and we report to NetworkManager the name of the interface to be used.
> All that works transparently for the user. If you switch to MM git
> master, all that should work correctly without additional changes in
> your side anywhere.
>
> > Now after this very long and convoluted story, my question is: Is there any particular fix on master branch, that could work for me, or is it possible to disable multiplexing explicitly? I think I could alter what's done in commit 05b005951710c8810538ec448e89e32f28c3d592 [3] to disable multiplexing entirely - as I don't need it currently. Cherry-picking this commit verbatim would not work, as cdc_mbim driver does support multiplexing, and currently, check is done only against mhi_net driver. Still, a fix regarding bearer cleanup would be much more appreciated, or a hint on where to start looking, so I could develop and upstream the fix.
>
> Quick workaround: disable IPv6 in the NM settings.
> Final solution: use MM git master as it is. Even if you don't need
> multiplexing, using it is straightforward, and all that should be
> correctly working (with any version of NetworkManager).
>
> > Thanks in advance for any information.
> >
> > [1] https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/issues/188 <https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/issues/188>
> > [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/cdc_mbim.rst?id=92f06f4226fd9bdd6fbbd2e8b84601fc14b5855e#n173 <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/cdc_mbim.rst?id=92f06f4226fd9bdd6fbbd2e8b84601fc14b5855e#n173>
> > [3] https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/commit/05b005951710c8810538ec448e89e32f28c3d592 <https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/commit/05b005951710c8810538ec448e89e32f28c3d592>
> >
>
> That was a long email; hope any of the things I said were helpful for you :)
>
> -- 
> Aleksander
> https://aleksander.es <https://aleksander.es>
>
> ___
>
> This email originated from outside of Camlin Group. Do not click on links or open attachments unless you are confident they are secure. If in doubt, contact IT Support (itsupport at camlintechnologies.com).
>
-- 
Pozdrawiam/With kind regards,
Lech Perczak

Sr. Software Engineer
Camlin Technologies Poland Limited Sp. z o.o.
Strzegomska 48A,
53-611 Wroclaw
Tel:     (+48) 71 75 000 16
Email:   l.perczak at camlintechnologies.com
Website: http://www.camlintechnologies.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/modemmanager-devel/attachments/20210621/2db23ec2/attachment-0001.htm>