<div dir="ltr">Hi.<div><br></div><div>I was mistaken about this ever happening with a 1500 MTU, it does not.</div><div>Looking into this further it seems to be an issue with the usbnet driver that qmi_wwan is leveraging. </div><div>Turning on debugfs and enabling debugging for usbnet.c shows the errors:</div><div><br></div><div>mount -t debugfs none /sys/kernel/debug<br>echo -n 'file usbnet.c +p' > /sys/kernel/debug/dynamic_debug/control<br></div><div><br></div><div>Looking at the log you can see that the rxqlen counter rapidly increases until it is full and a throttle occurs, this throttle error then repeats forever (quite rapidly) until the modem is reset, even if the ping is stopped:</div><div><br></div><div><font face="monospace">Sep 23 10:16:16 T6 kernel: [ 715.615456] qmi_wwan 2-2:1.8 wwan0: rx throttle -71<br>Sep 23 10:16:16 T6 kernel: [ 715.743152] qmi_wwan 2-2:1.8 wwan0: rxqlen 281 --> 291<br>Sep 23 10:16:16 T6 kernel: [ 715.743307] qmi_wwan 2-2:1.8 wwan0: rxqlen 291 --> 301<br>Sep 23 10:16:16 T6 kernel: [ 715.743320] qmi_wwan 2-2:1.8 wwan0: rxqlen 301 --> 311<br>Sep 23 10:16:16 T6 kernel: [ 715.743331] qmi_wwan 2-2:1.8 wwan0: rxqlen 311 --> 318<br>Sep 23 10:16:16 T6 kernel: [ 715.744947] qmi_wwan 2-2:1.8 wwan0: rx throttle -71<br>Sep 23 10:16:16 T6 kernel: [ 715.871077] qmi_wwan 2-2:1.8 wwan0: rxqlen 281 --> 291<br>Sep 23 10:16:16 T6 kernel: [ 715.871115] qmi_wwan 2-2:1.8 wwan0: rxqlen 291 --> 301<br>Sep 23 10:16:16 T6 kernel: [ 715.871128] qmi_wwan 2-2:1.8 wwan0: rxqlen 301 --> 311<br>Sep 23 10:16:16 T6 kernel: [ 715.871138] qmi_wwan 2-2:1.8 wwan0: rxqlen 311 --> 318<br>Sep 23 10:16:16 T6 kernel: [ 715.874448] qmi_wwan 2-2:1.8 wwan0: rx throttle -71<br>Sep 23 10:16:17 T6 kernel: [ 716.007133] qmi_wwan 2-2:1.8 wwan0: rxqlen 280 --> 290<br>Sep 23 10:16:17 T6 kernel: [ 716.007172] qmi_wwan 2-2:1.8 wwan0: rxqlen 290 --> 300<br>Sep 23 10:16:17 T6 kernel: [ 716.007185] qmi_wwan 2-2:1.8 wwan0: rxqlen 300 --> 310<br>Sep 23 10:16:17 T6 kernel: [ 716.007197] qmi_wwan 2-2:1.8 wwan0: rxqlen 310 --> 318</font><br></div><div><font face="monospace"><br></font></div><div><br></div><div><font face="arial, sans-serif">This happens on USB3 and not USB2 (unless I pick a small MTU like 500). With USB2 I generally don't get any of the above errors and can then go back to pinging with regular sized packets no problem.</font></div><div><font face="arial, sans-serif">The above error, 71 refers to: </font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="monospace">#define EPROTO 71 /* Protocol error */</font><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif"><br></font></div><div><p style="margin:10px 0px 0px;padding:0px;color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)">The rxqlen error is coming from usbnet_bh () which is described as <em>“tasklet (work deferred from completions, in_irq) or timer”</em> in the source code. This is too large to paste, but looks like the cleanup function to release the Socket Buffers (skb) and USB Request Blocks (urb). Interesting snippet of code from this function, after releasing the buffers:</p><div class="gmail-code gmail-panel" style="margin:9px 0px;padding:0px;border:1px solid rgb(204,204,204);background:rgb(245,245,245);font-size:12px;line-height:1.33333;font-family:monospace;border-radius:3px;color:rgb(23,43,77)"><div class="gmail-codeContent gmail-panelContent" style="margin:0px;padding:9px 12px"><pre class="gmail-code-c" style="margin-top:0px;margin-bottom:0px;padding:0px;max-height:30em;overflow:auto;white-space:pre-wrap"> <span class="gmail-code-comment" style="color:rgb(128,128,128)">/* restart RX again after disabling due to high error rate */</span> clear_bit(EVENT_RX_KILL, &dev->flags);</pre></div></div><p style="margin:10px 0px 0px;padding:0px;color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)">Buffers are coming in with -EPROTO at a high rate so the usbnet driver disables RX. The buffers get released, which explains the packet loss, and Rx is restarted. Obviously whatever issue was underlying is still present and so Rx is disabled again and the dance restarts.</p><p style="margin:10px 0px 0px;padding:0px;color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)">There are a few scenarios it can occur, but the two easiest to reproduce are:</p><ol style="margin:10px 0px 0px;color:rgb(23,43,77);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,"Fira Sans","Droid Sans","Helvetica Neue",sans-serif;font-size:14px;background-color:rgb(244,245,247)"><li>Send pings larger than the non-1500 MTU, and they occur after ~300seconds.</li><li>Set a non 1500 MTU. reset the modem without sending anything, and they occur immediately afterwards.</li></ol></div><div><br></div><div><font face="arial, sans-serif">Still trying to figure it out,</font></div><div><font face="monospace"><br></font></div><div><font face="arial, sans-serif">--</font></div><div><font face="arial, sans-serif">Paul</font></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 24 Jan 2020 at 17:26, Paul Gildea <<a href="mailto:gildeap@tcd.ie">gildeap@tcd.ie</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi guys, have done some more testing with this. Using ubuntu 16.04 I installed libqmi (1.16) to test that also, on a fresh system (a laptop). <div>Set the MTU of the private network to 1430 and pinged with large packets (2000) and after a couple of hundred pings this behaviour repeated itself.</div><div>When I thought that the system was fine yesterday, it turns out it just took more pings for it to fall over. At 1500 MTU and 300 pings the same thing occurred with the private network.</div><div>Massive amounts of input errors, modem becomes unusable until reset.</div><div><br></div><div>I can see pings arriving at the back end, it's when they are returning to the modem that there appears to be an issue. </div><div>This is happening with SW and Telit with varying modems and multiple networks. </div><div>It seems to me to be a qmi_wwan driver issue, what do you think?</div><div><br></div><div>Regards</div><div><br></div><div>--</div><div>Paul</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 23 Jan 2020 at 17:15, Paul Gildea <<a href="mailto:gildeap@tcd.ie" target="_blank">gildeap@tcd.ie</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi, </div><div><br></div><div>We are seeing MTU issues with the ATT network and were just wondering if you had heard of anything like this happening before? ATT push an MTU of 1430, we apply that to our linux interface and everything works fine. However when we try and ping with a large packet and the pings fail instead of fragmenting correctly, we also see input errors on the linux interface.</div><div><br></div><div>In some scenarios the input errors never stop (getting thousands of them every second) after we stop pinging and the modem needs to be rebooted to pass any traffic again. Changed MTU values to a lot of different things and this always occurs.</div><div><br></div><div>On a private network here (MTU 1404) and Vodafone (MTU 1500) doing the same thing with the network pushed MTU values causes no issues and everything works fine. I tested moving away from 1500 on vodafone to lower values and once the issues reoccurred, same as with ATT.</div><div><br></div><div>Libqmi 1.24 and I have seen this behaviour with all modems tested so far: MC7455, EM7565, EM7511 and Telit LM960A18.</div><div><br></div><div>Regards,</div><div><br></div><div>--</div><div>Paul</div></div> </blockquote></div> </blockquote></div>