EM7565 failing if host reboots whithout power cycling the modem

[Bjørn Mork]

John Marrett <johnf at zioncluster.ca> writes:

>> But what can be done about that?  From my point of view, fixing the
>> issue in the modem firmware is out of the question. The modem is running
>> SWI9X50C_01.14.13.00, which the latest available.
>>
>
> You should either open a case with the modem manufacturer if you have some
> kind of support agreement.

You're right, of course. This modem was graciuosly dumped in my mailbox
by one of the mailing list participants a few years ago, and I suspect
that I could follow the chain back for support.

But I don't have enough faith in that solution to give it a chance.
Sorry.  I'll try to explain.

I used to get great support for my MC7710 from my local FAE a decade
ago. But it was limited to what they could dig up of documentation and
existing firmware.  Obvious, simple, spec violating and stupid firmware
bugs like MBIM functions without any CDC Union descriptor was never
fixed.  So we had to work around the bug in the kernel instead.

And talking about Qualcomm bugs: The final nail was when Sierra Wireless
included one of my workarounds in their GobiNet driver.  They even
copied my verbose explanation of the firmware bug, with spelling errors
and all :-)

>From the Sierra Wireless GobiUSBNet.c:

/* Make up an ethernet header if the packet doesn't have one.
 *
 * A firmware bug common among several devices cause them to send raw
 * IP packets under some circumstances.  There is no way for the
 * driver/host to know when this will happen.  And even when the bug
 * hits, some packets will still arrive with an intact header.
 *
 * The supported devices are only capably of sending IPv4, IPv6 and
 * ARP packets on a point-to-point link. Any packet with an ethernet
 * header will have either our address or a broadcast/multicast
 * address as destination.  ARP packets will always have a header.
 *
 * This means that this function will reliably add the appropriate
 * header iff necessary, provided our hardware address does not start
 * with 4 or 6.
 *
 * Another common firmware bug results in all packets being addressed
 * to 00:a0:c6:00:00:00 despite the host address being different.
 * This function will also fixup such packets.
 */
static int GobiNetDriverLteRxFixup(struct usbnet *dev, struct sk_buff *skb)
{
[etc]

Compare that to
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/usb/qmi_wwan.c#n543

Funny, isn't it? :-)

Anyway.  That was a pretty clear signal to me that there is no point
reporting any Qualcomm firmware bugs to Sierra Wireless.  They are
incapable of getting them fixed, for whatever reason.  So much so that
they prefer extremely hackish workarounds from the community instead.

This isn't criticizing Sierra Wireless in any way.  I believe these
problems are entirely within Qualcomm.  All Qualcomm customers are
struggling with the same.

> If you don't then you could ask about it on the
> sierra wireless forums ( https://forum.sierrawireless.com/ ). This problem
> is likely to affect other people and you have very detailed and clean
> information about the problem. Sierra can raise it to the modem
> manufacturer.

Yes.  I enjoy those forums and did consider mentioning the problem
there. Probably should.  It is a more appropriate place for firmware
related isues.

> Have you tried unloading xhci_pci before rebooting the system? This might
> help?

Tried that now, and it did help.  Or...  Just double checked by
rebooting again without unloading xhci_pci, and that also worked.  So
much for consistent behaviour.  I don't know why, but it seems that the
issue is gone for now.  Unfortunately, I must say. Just makes it harder
to track down.

>
>> We need a woraround for the host system.  But what?  Since I have adb
>> access, I can obviously just reboot the modem.  Which works.  But what
>> about the "normal" modem configuration?  Figure out a way to cut USB
>> port power?  I guess this is possible on that platform, although I don't
>> think it's implemented.  It's not a very generic solution, though.  But
>> maybe this isn't a very generic problem?
>
>
> Unfortunately this is a very normal problem :) I manage a lot of modems of
> different types and models and also do some consulting in this space. On
> every platform and with a variety of modems it's always been necessary to
> implement watchdog scripts that reboot the modem when it fails to stop
> responsing. I've always found a GPIO to cut power to the modem in a worst
> case. I don't see a named usb gpio in your router's OpenWrt configuration
> but I expect that there must be something, even if it isn't named. I'm
> happy to help you explore /sys/class/gpio if you need a hand.

Oh, I don't think there is anything defined.  I was more thinking of
looking at the regulators and power domains and see if I could add the
necessary control knob.

>> Maybe it's just this exact
>> combo of host controller, device and continous power to the port during
>> reboot (which I'm very much in favour of in general)?
>>
>> Have anyone else noticed similar problems with the EM7565?  Or any other
>> Qualcomm MDM9x50 device?  I assume this bug isn't really Sierra Wireless
>> specific, but rather something they got from Qualcomm.
>>
>
> Yes, but Sierra will raise it to Qualcomm.
>
>
>> You'll obviously not see this problem if your host system cuts power to
>> the USB port on reboots.  Which unfortunately often is the case.
>>
>
> I haven't seen this kind of behaviour before. USB devices have always been
> rebooted with the host on other embedded routers.

Yes, I know. Almost all host devices will cut power to USB ports during
reboot.  And hosts with m.2 or mini PCIs slots will usually assert the
reset lines too.  The continuously powered USB port is an exception,
making my configuration special.  But it should still work.


Bjørn