High memory usage by ModemManager

Mon Nov 25 10:40:10 UTC 2019

Hey,

>
> We are using ModemManager with the Quectel BG96 modem and are using it's QMI interface.
> Everything is going well, expect that ModemManager starts to use a lot of the memory after it is running for a while.
>

That's unexpected, MM shouldn't start using lots of memory unless
there's some obvious memory leak somewhere.

> The Quectel BG96 modem has been resetted 3 times during its operation by sending "AT+CFUN=1,1" to the primary AT interface.
> This is done, by a script we have running on the device, when the connection is lost for 5 minutes.

How do you do this? Are you talking to the TTY port directly, or are
you running CFUN=1,1 through mmcli?
Have you tried to run mmcli -m X --reset instead?

> After the modem is reset, the script checks if the ModemManager can see the modem and will check the GPS settings of the modem. This is done via the primary AT interface.

Again, how? via mmcli? or via the TTY port directly?

> After this, we turn on the GPS via ModemManager using the Location interface.
>
> Commands used for enabling GPS ( $modem_id is requested from DBus ):
> mmcli -m $modem_id --location-enable-gps-nmea
> mmcli -m $modem_id --location-enable-gps-raw
> mmcli -m $modem_id --location-set-enable-signal
> mmcli -m $modem_id--location-set-gps-refresh-rate=0
>
> When this is done, we setup the cellular connection using NetworkManager.
>

Ok.

> Currently we see that this has produced ModemManager running 3 instances of itself and using 25% of the 512MB of memory.

This is not truly possible; there can only be one ModemManager daemon
running at a time, because only one process can acquire the well-known
DBus name in the system bus. I have really never seen more than 1
instance of MM running at the same time. Why do you say there are 3?
do you really see 3 ModemManager processes with 3 different PIDs all
running at the same time?

> What also happens is that the GPS is not always turned on after a modem reset.
>
> Expected state:
>
> $ mmcli -m 0 --location-status
> --------------------------------
> Location |  capabilities: 3gpp-lac-ci, gps-raw, gps-nmea, agps
> | enabled: 3gpp-lac-ci, gps-raw, gps-nmea
> | signals: yes
> --------------------------------
> GPS | refresh rate: 0 seconds
>
> Actual state:
>
> $ mmcli -m 0 --location-status
> --------------------------------
> Location |  capabilities: 3gpp-lac-ci, gps-raw, gps-nmea, agps
> |       enabled: 3gpp-lac-ci
> |        signals: no
> -------------------------------
> GPS | refresh rate: 30 seconds

This could be because MM couldn't grab the QMI interface for some
reason. There is one QMI interface usually, right? We would need to
see MM debug logs when this happens to understand why the QMI port
probing failed.

>
> Are we doing something wrong in the way we are resetting the modem after a long duration of connection loss?
>

Depends on how you talk to the TTY port to run the CFUN command
really. If you could try with mmcli --reset, that may also give good
results to do so. Although, ideally, there shouldn't be any reason why
you need to reset the modem :)

> Do we need to restart ModemManager each time we reset the modem to prevent multiple instances of ModemManager to be running at the same time?
>

Need more information about this issue, because there's no way this
can happen really.

> What we also see is that ModemManager is giving a lot of warnings, does this has something to do with the problems we are experiencing?
>
> Logs from ModemManager:
>
> Nov 25 07:04:39 ModemManager[158]: <warn>  Reloading stats failed: QMI operation failed: Transaction timed out
> Nov 25 07:04:41 ModemManager[158]: transaction 0x26e aborted, but message is not abortable
> Nov 25 07:04:59 ModemManager[158]: [/dev/cdc-wdm0] No transaction matched in received message
> Nov 25 07:04:59 ModemManager[158]: [/dev/cdc-wdm0] No transaction matched in received message
> Nov 25 07:05:09 ModemManager[158]: transaction 0x26f aborted, but message is not abortable
> Nov 25 07:05:09 ModemManager[158]: <warn>  Reloading stats failed: QMI operation failed: Transaction timed out
> Nov 25 07:06:39 ModemManager[158]: transaction 0x272 aborted, but message is not abortable
> Nov 25 07:06:39 ModemManager[158]: <warn>  Reloading stats failed: QMI operation failed: Transaction timed out
> Nov 25 07:06:41 ModemManager[158]: [/dev/cdc-wdm0] No transaction matched in received message
> Nov 25 07:49:28 ModemManager[158]: transaction 0x2c7 aborted, but message is not abortable
> Nov 25 08:05:09 ModemManager[158]: transaction 0x2e7 aborted, but message is not abortable
> Nov 25 08:05:09 ModemManager[158]: <warn>  Reloading stats failed: QMI operation failed: Transaction timed out
> Nov 25 08:09:39 ModemManager[158]: transaction 0x2f0 aborted, but message is not abortable
> Nov 25 08:09:39 ModemManager[158]: <warn>  Reloading stats failed: QMI operation failed: Transaction timed out
> Nov 25 08:10:39 ModemManager[158]: transaction 0x2f2 aborted, but message is not abortable
> Nov 25 08:10:39 ModemManager[158]: <warn>  Reloading stats failed: QMI operation failed: Transaction timed out
> Nov 25 08:29:39 ModemManager[158]: transaction 0x318 aborted, but message is not abortable
> Nov 25 08:29:39 ModemManager[158]: <warn>  Reloading stats failed: QMI operation failed: Transaction timed out
> Nov 25 08:45:09 ModemManager[158]: transaction 0x337 aborted, but message is not abortable
> Nov 25 08:45:09 ModemManager[158]: <warn>  Reloading stats failed: QMI operation failed: Transaction timed out
> Nov 25 09:03:39 ModemManager[158]: transaction 0x35c aborted, but message is not abortable
> Nov 25 09:03:39 ModemManager[158]: <warn>  Reloading stats failed: QMI operation failed: Transaction timed out
> Nov 25 09:06:09 ModemManager[158]: transaction 0x361 aborted, but message is not abortable
> Nov 25 09:06:09 ModemManager[158]: <warn>  Reloading stats failed: QMI operation failed: Transaction timed out
>

Those timeouts could indicate that the QMI port is not responding. Are
you using the QMI port through qmicli yourselves in any way?

> Version of ModemManager used: 1.10.8
> Version of libqmi used: 1.24.0

You should update to MM 1.12! :)

-- 
Aleksander
https://aleksander.es