Random MM and/or qmi-proxy hang?

[Bowden, Brendan]

> From: Aleksander Morgado <aleksander at aleksander.es>
> Sent: Wednesday, March 13, 2019 6:36 AM
> To: Bowden, Brendan <bbowden at presidio.com>
> Cc: ModemManager (development) <modemmanager-
> devel at lists.freedesktop.org>
> Subject: Re: Random MM and/or qmi-proxy hang?
>
>
> Hey,
>
> >
> > This is RHEL7 so I’m using the 1.6 branch of MM and 1.18.0 libqmi; for MM I
> tried building our own 1.16.4 when we suspected MM being the core issue,
> with no change in behavior.
> >
>
> Hard to say what the issue is, but you're really using somewhat old versions.
> Any chance you can upgrade them to MM 1.10.0 and libqmi 1.22.2? They
> should be totally compatible.

Upgraded to MM 1.10, libqmi 1.22.2. Definitely helped; not getting the qmi-proxy lock-ups anymore. But I think it ended up exposing what our actual problem is: we're now seeing ClientIdsExhausted errors. The timing would suggest this is happening where we were locking up before.

At this point I should mention we have a custom app also speaking direct libqmi to both modems to extract detailed per-chain RF data every minute or so (qmi_message_nas_get_tx_rx_info_output_get_rx_chain_0_info and the same on chain_1). It follows a process of connect to qmi-proxy with QMI_CID_NONE set (to allocate a new CID), register to DMS and NAS services, gather data, then unregister/deallocate. After doing this enough times we're apparently running out of CIDs even though we’re deallocating each time. Tried the qmicli 'sync' function to flush out the modem and that very much confused MM, the modem disconnects and gets stuck in an 'unavailable' state. Mmcli modem rescan doesn't help at that point, have to power cycle the modem, which now that we've lost QMI access can only be done via AT console or whole machine power-cycle.

Any way to get these per-chain signal values via MM so we don't have to talk to the modems directly?
If not, any idea what we might be doing wrong with our QMI interactions? We're already looking into keeping one QMI client per modem up front and reusing it until the modem disappears, but that will take a major refactor of the code so hoping we don't have to go that far.

Thanks!


>
> >
> > Any thoughts? Wanted to see about putting qmi-proxy in debug mode but
> I’m not sure how to do that when it’s already running (spawned by MM). Out
> of ~200 units I’ve got MM debug logging enabled on, 3-4 would get into this
> state after 24 hours, so reproducibility is a problem; haven’t found a way to
> trigger this condition on cue yet. As background info, each unit (3500 out
> there) has two Sierra modems, could be any combination of MC7700,
> MC7750, MC7354 talking to AT&T or Verizon, both LTE and 3G.
>
> You can run the qmi-proxy with verbose info by manually starting it BEFORE
> MM is started.
> E.g.
> $ sudo systemctl stop ModemManager
> $ sudo /usr/libexect/qmi-proxy --no-exit --verbose > /tmp/qmi-proxy-
> verbose.log 2>&1 & $ sudo systemctl start ModemManager
>
>
> --
> Aleksander
> https://aleksander.es


This message w/attachments (message) is intended solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments. Please be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited.