E173s-1 stops working after weeks of uptime
dirbaio at dirbaio.net
Thu May 30 18:49:15 UTC 2019
We have some embedded devices deployed in the field using a huawei E173s-1
modem. We're having some issues where connectivity stops working randomly,
after 1-3 weeks of uptime.
First, there's a PPP disconnection. Then there's a few loops of this error
for 1-2 seconds, rather fast:
failed to connect modem: Couldn't connect: cannot keep data port
open.Could not open serial device ttyUSB0: reopen operation in progress
Afterwards, there are 10 attempts of reconnecting, with "at port timed out
X consecutive times" errors. After 10 attempts, ModemManager gives up
permanently, and never tries again to bring the device back up.
(tty/ttyUSB0) at port timed out 10 consecutive times, marking modem
'/org/freedesktop/ModemManager1/Modem/0' as invalid
A few hours later, when we could get onsite, restarting ModemManager and
NetworkManager brought the modem back online with no issues (with no
unplug/replug/powercycle of the modem or the Linux board.) This means the
modem is not irreversibly crashed/failed, so I think this is a software
issue that should be fixable.
Unfortunately, ModemManager was set to INFO log level because we though
DEBUG was too verbose. We have set DEBUG log level, so the next time it
happens we will have more logs.
I would really appreciate any input you may have on how to solve this.
- I thought of patching out the "max 10 timeouts" limit, so ModemManager
keeps retrying indefinitely. Is this a good idea?
- What can I try so next time this happens we can get more info on the
issue? (besides debug log level)
- Any recommendations in general on how to ensure the device is always
connected? Any config knobs to tweak? We've been thinking of adding a "if
no internet during 10 minutes, powercycle everything" watchdog, but that
feels like giving up on getting this working properly.
Thanks in advance!
Logs of the time of failure:
Output of mmcli -m 0:
Relevant software versions:
- ModemManager 1.10.0
- NetworkManager 1.16.0
- Linux 5.0.5
- pppd 2.4.7
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the ModemManager-devel