E173s-1 stops working after weeks of uptime

[Dario Nieuwenhuis]

Hello,

We have some embedded devices deployed in the field using a huawei E173s-1
modem. We're having some issues where connectivity stops working randomly,
after 1-3 weeks of uptime.

First, there's a PPP disconnection. Then there's a few loops of this error
for 1-2 seconds, rather fast:

    failed to connect modem: Couldn't connect: cannot keep data port
open.Could not open serial device ttyUSB0: reopen operation in progress

Afterwards, there are 10 attempts of reconnecting, with "at port timed out
X consecutive times" errors. After 10 attempts, ModemManager gives up
permanently, and never tries again to bring the device back up.

    (tty/ttyUSB0) at port timed out 10 consecutive times, marking modem
'/org/freedesktop/ModemManager1/Modem/0' as invalid

A few hours later, when we could get onsite, restarting ModemManager and
NetworkManager brought the modem back online with no issues (with no
unplug/replug/powercycle of the modem or the Linux board.) This means the
modem is not irreversibly crashed/failed, so I think this is a software
issue that should be fixable.

Unfortunately, ModemManager was set to INFO log level because we though
DEBUG was too verbose. We have set DEBUG log level, so the next time it
happens we will have more logs.

I would really appreciate any input you may have on how to solve this.
- I thought of patching out the "max 10 timeouts" limit, so ModemManager
keeps retrying indefinitely. Is this a good idea?
- What can I try so next time this happens we can get more info on the
issue? (besides debug log level)
- Any recommendations in general on how to ensure the device is always
connected? Any config knobs to tweak? We've been thinking of adding a "if
no internet during 10 minutes, powercycle everything" watchdog, but that
feels like giving up on getting this working properly.

Thanks in advance!

Dario

---

Logs of the time of failure:
https://gist.github.com/Dirbaio/bdea5235b832ee12d4b56caf7576fe3d
Output of mmcli -m 0:
https://gist.github.com/Dirbaio/13c97d9ec508faba0829ce9de0cf79de
lsusb output:
https://gist.github.com/Dirbaio/0e58a51fd551272f92530f96c612b8c6

Relevant software versions:
- ModemManager 1.10.0
- NetworkManager 1.16.0
- Linux 5.0.5
- pppd 2.4.7
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/modemmanager-devel/attachments/20190530/eb8c31b9/attachment.html>