E173s-1 stops working after weeks of uptime

[Dan Williams]

On Fri, 2019-05-31 at 11:18 +0200, Ladislav Michl wrote:
> On Thu, May 30, 2019 at 09:46:01PM -0500, Dan Williams wrote:
> > On Fri, 2019-05-31 at 00:01 +0200, Ladislav Michl wrote:
> [...]
> ...and I'll eventually create new thread to not continue hikacking
> this one.
> > > I'm using MM&NM restart and if that doesn't help, then modem
> > > powercycle and
> > > then daemon restart. Boards do have modem power supply controlled
> > > by
> > > gpio,
> > > so that's easy to do, but rather hackish. Is there any plan to
> > > add
> > > some
> > > "modem hw reset" infrastructure here?
> > 
> > We've thought about it before, but the problem is that it's pretty
> > unreliable for USB ports on most machines that aren't embedded.
> > Basically, you cannot guarantee that the USB host controller
> > supports
> > the command to power cycle a port, and a lot of them just don't.
> > 
> > You can't expect the USB port reset to work, because that's a USB
> > request to the firmware running on the device IIRC and if the
> > device is
> > already hung it's surely not going to pay attention to a reset
> > request.
> 
> Even worse, many quectels have nRST and PWR_KEY pins which name
> suggests
> to reset or power down/up them, but those are just gpios for them, so
> the
> only reliable way to reset is power cycle. And most of sane hardware
> designers are creating their hardware with that in mind.
> 
> > So basically yes, that infrastructure could be added, but there's
> > no
> > way you can depend on the request actually succeeding especially on
> > x86
> > machines.
> 
> I'm not talking about making it anyhow mandatory and not even about
> ordinary laptops or PCs with frustrated user always able to replug
> poor
> USB modem once their favourite web page stops loading.
> 
> ModemManager hit OpenWrt recently and is in PTXdist for ages, so
> number
> of "embedded" instalations could easily overgrow those "ordinary"
> ones.
> And each that embedded device have to deal with unreliable modem
> somehow,
> so I'd say it could be usefull to have at least some recomendation
> how to handle such powercycles.

Yes, definitely useful there if the platform has a way of actually
killing power to the device, whether that's USB or GPIO or whatever.
And I think it would be fine to add a method to the Modem object to do
this.

If there's a lot of variability in platforms (and I think there is :)
we could theoretically have MM call out to a script to poke GPIOs or
sysfs or run some other program.

A start would be gathering the data on the different ways that
platforms allow killing power to modems. TO start with, if it's USB and
the hub reports per-port power control in wHubCharacteristic via lsusb
-v then we might be able to power cycle.

Dan

> The thing is that those devices often do not have enough storage to
> run
> MM with debug log level, are otherwise unreachable, their CPU is
> often
> less powerfull than those used in modems (which could teoretically
> run
> all that software running on the board they are plugged into, but
> that's
> different story) and hangs does not occur so often, to be reasonably
> debuggable. All that makes modem device power cycling the only
> reliable
> way to recover in field (except whole machine power on reset, of
> course)