<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi!</div><div dir="ltr"><br></div><div dir="ltr">I couldn't figure out what to log in mhi_init_irq_setup but i put a logline on in mhi_init_dev_ctxt:</div><div dir="ltr"><div dir="ltr">oskar@oskar-ThinkPad-X1-Nano-Gen-1:~/src/linux-kernel/linux-upstream-5.12.0+/drivers/bus/mhi$ git diff .</div><div dir="ltr">diff --git a/drivers/bus/mhi/core/init.c b/drivers/bus/mhi/core/init.c</div><div dir="ltr">index c81b377fca8f..732affde7a20 100644</div><div dir="ltr">--- a/drivers/bus/mhi/core/init.c</div><div dir="ltr">+++ b/drivers/bus/mhi/core/init.c</div><div dir="ltr">@@ -266,6 +266,7 @@ int mhi_init_dev_ctxt(struct mhi_controller *mhi_cntrl)</div><div dir="ltr"> struct mhi_cmd *mhi_cmd;</div><div dir="ltr"> u32 tmp;</div><div dir="ltr"> int ret = -ENOMEM, i;</div><div dir="ltr">+ struct device *dev = &mhi_cntrl->mhi_dev->dev;</div><div dir="ltr"><br></div><div dir="ltr"> atomic_set(&mhi_cntrl->dev_wake, 0);</div><div dir="ltr"> atomic_set(&mhi_cntrl->pending_pkts, 0);</div><div dir="ltr">@@ -333,6 +334,9 @@ int mhi_init_dev_ctxt(struct mhi_controller *mhi_cntrl)</div><div dir="ltr"><br></div><div dir="ltr"> er_ctxt->ertype = MHI_ER_TYPE_VALID;</div><div dir="ltr"> er_ctxt->msivec = mhi_event->irq;</div><div dir="ltr">+ dev_warn(dev, "er_ctxt->msivec::%d\n",</div><div dir="ltr">+ er_ctxt->msivec);</div><div dir="ltr">+</div><div dir="ltr"> mhi_event->db_cfg.db_mode = true;</div><div dir="ltr"><br></div><div dir="ltr"> ring->el_size = sizeof(struct mhi_tre);</div><div><br></div><div><br></div><div>With intel vt-d disabled:</div><div><div>[ 64.428320] mhi: loading out-of-tree module taints kernel.</div><div>[ 64.428455] mhi: module verification failed: signature and/or required key missing - tainting kernel</div><div>[ 98.714766] mhi-pci-generic 0000:08:00.0: BAR 0: assigned [mem 0xbc200000-0xbc200fff 64bit]</div><div>[ 98.714900] mhi-pci-generic 0000:08:00.0: enabling device (0000 -> 0002)</div><div>[ 98.715245] mhi-pci-generic 0000:08:00.0: using shared MSI</div><div>[ 98.715611] mhi mhi0: er_ctxt->msivec::0</div><div>[ 98.715614] mhi mhi0: er_ctxt->msivec::0</div><div>[ 98.715617] mhi mhi0: er_ctxt->msivec::0</div><div>[ 98.715623] mhi mhi0: er_ctxt->msivec::0</div><div>[ 98.715631] mhi mhi0: Requested to power ON</div><div>[ 98.716156] mhi mhi0: Power on setup success</div><div>[ 98.750461] mhi mhi0: Wait for device to enter SBL or Mission mode</div></div><div><br></div><div>With intel vt-d enabled:</div><div><div>[ 64.750646] mhi: loading out-of-tree module taints kernel.</div><div>[ 64.750776] mhi: module verification failed: signature and/or required key missing - tainting kernel</div><div>[ 71.229020] mhi-pci-generic 0000:08:00.0: BAR 0: assigned [mem 0xbc200000-0xbc200fff 64bit]</div><div>[ 71.229153] mhi-pci-generic 0000:08:00.0: enabling device (0000 -> 0002)</div><div>[ 71.230017] mhi mhi0: er_ctxt->msivec::1</div><div>[ 71.230021] mhi mhi0: er_ctxt->msivec::2</div><div>[ 71.230024] mhi mhi0: er_ctxt->msivec::3</div><div>[ 71.230030] mhi mhi0: er_ctxt->msivec::4</div><div>[ 71.230038] mhi mhi0: Requested to power ON</div><div>[ 71.231107] mhi mhi0: Power on setup success</div><div>[ 71.264612] mhi mhi0: Wait for device to enter SBL or Mission mode</div></div><div><br></div><div>Do i understand it correctly that this is correctly telling the modem to use shared irq and that this might be a modem firmware-bug?</div><div><br></div><div>/Oskar Stenman</div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 18 May 2021 at 12:16, Loic Poulain <<a href="mailto:loic.poulain@linaro.org" target="_blank">loic.poulain@linaro.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Oskar,<br><br>On Tue, 18 May 2021 at 11:33, Oskar Stenman <<a href="mailto:oskar@cetex.se" target="_blank">oskar@cetex.se</a>> wrote:<br>><br>> Hi!<br>><br>> I'm way out in the deep end of the pool here poking at things i shouldn't to see what happens and if i can learn something, or find a bit more info where things go wrong.. So I patched drivers/bus/mhi/pci_generic.c to give some more debug info:<br>><br>> oskar@oskar-ThinkPad-X1-Nano-Gen-1:~/src/linux-kernel/linux-upstream-5.12.0+$ git diff<br>> diff --git a/drivers/bus/mhi/pci_generic.c b/drivers/bus/mhi/pci_generic.c<br>> index 7c810f02a2ef..819c0f17993d 100644<br>> --- a/drivers/bus/mhi/pci_generic.c<br>> +++ b/drivers/bus/mhi/pci_generic.c<br>> @@ -492,9 +492,13 @@ static int mhi_pci_get_irqs(struct mhi_controller *mhi_cntrl,<br>> * Alloc one MSI vector for BHI + one vector per event ring, ideally...<br>> * No explicit pci_free_irq_vectors required, done by pcim_release.<br>> */<br>> + dev_warn(&pdev->dev, "mhi_cntrl_config->num_events: %d\n", mhi_cntrl_config->num_events);<br>> mhi_cntrl->nr_irqs = 1 + mhi_cntrl_config->num_events;<br>> + dev_warn(&pdev->dev, "mhi_cntrl->nr_irqs: %d\n", mhi_cntrl->nr_irqs);<br>> + dev_warn(&pdev->dev, "PCI_IRQ_MSI: %d\n", PCI_IRQ_MSI);<br>><br>> nr_vectors = pci_alloc_irq_vectors(pdev, 1, mhi_cntrl->nr_irqs, PCI_IRQ_MSI);<br>> + dev_warn(&pdev->dev, "nr_vectors: %d\n", nr_vectors);<br>> if (nr_vectors < 0) {<br>> dev_err(&pdev->dev, "Error allocating MSI vectors %d\n",<br>> nr_vectors);<br>><br>> Compiling and loading that module prints the following in dmesg:<br>> [ 5553.096754] mhi-pci-generic 0000:08:00.0: BAR 0: assigned [mem 0xbc200000-0xbc200fff 64bit]<br>> [ 5553.097033] mhi-pci-generic 0000:08:00.0: mhi_cntrl_config->num_events: 4<br>> [ 5553.097036] mhi-pci-generic 0000:08:00.0: mhi_cntrl->nr_irqs: 5<br>> [ 5553.097038] mhi-pci-generic 0000:08:00.0: PCI_IRQ_MSI: 2<br>> [ 5553.097314] mhi-pci-generic 0000:08:00.0: nr_vectors: 1 <- requested 5, got 1..<br>> [ 5553.097316] mhi-pci-generic 0000:08:00.0: using shared MSI<br>><br>> requesting 5 nr_vectors, getting one which then seems to trigger another codepath..<br>><br>> Whlie researching pci_alloc_irq_vectors I then found this semi-related thread <a href="https://stackoverflow.com/questions/49821599/multiple-msi-vectors-linux-pci-alloc-irq-vectors-return-one-while-the-devi" target="_blank">https://stackoverflow.com/questions/49821599/multiple-msi-vectors-linux-pci-alloc-irq-vectors-return-one-while-the-devi</a> where Riton38 mentions that enabling intel vt-d "solved the problem" (default from factory on this Thinkpad X1 Nano it has all virtualization disabled).<br>><br>> Since i can't figure out what's wrong with the actual irq-setup as it's possibly bug in modem firmware (throwing interrupts over some channel when it shouldn't), or maybe a bug in how irq's are allocated in mhi_pci_generic i just tried enabling Intel VT-D in bios (security -> virtualization) and what would you know, it works!!<br>><br>> [ 81.291874] mhi_pci_generic: loading out-of-tree module taints kernel.<br>> [ 81.291970] mhi_pci_generic: module verification failed: signature and/or required key missing - tainting kernel<br>> [ 81.292527] mhi-pci-generic 0000:08:00.0: BAR 0: assigned [mem 0xbc200000-0xbc200fff 64bit]<br>> [ 81.292560] mhi-pci-generic 0000:08:00.0: enabling device (0000 -> 0002)<br>> [ 81.292603] mhi-pci-generic 0000:08:00.0: mhi_cntrl_config->num_events: 4<br>> [ 81.292605] mhi-pci-generic 0000:08:00.0: mhi_cntrl->nr_irqs: 5<br>> [ 81.292606] mhi-pci-generic 0000:08:00.0: PCI_IRQ_MSI: 2<br>> [ 81.292713] mhi-pci-generic 0000:08:00.0: nr_vectors: 5 <- We got 5 vectors now!<br>> [ 81.292955] mhi mhi0: Requested to power ON<br>> [ 81.293180] mhi mhi0: Power on setup success<br>> [ 81.325854] mhi mhi0: Wait for device to enter SBL or Mission mode<br>><br>> After running qmicli and connecting, no more errors in kernel log.<br>><br>> oskar@oskar-ThinkPad-X1-Nano-Gen-1:~/src/linux-kernel/linux-upstream-5.12.0+/drivers/bus/mhi$ ping <a href="http://ftp.sunet.se" target="_blank">ftp.sunet.se</a><br>> PING <a href="http://ftp.sunet.se" target="_blank">ftp.sunet.se</a>(<a href="http://tutankhamon.ftp.acc.umu.se" target="_blank">tutankhamon.ftp.acc.umu.se</a> (2001:6b0:19::163)) 56 data bytes<br>> 64 bytes from <a href="http://tutankhamon.ftp.acc.umu.se" target="_blank">tutankhamon.ftp.acc.umu.se</a> (2001:6b0:19::163): icmp_seq=1 ttl=51 time=46.1 ms<br>> 64 bytes from <a href="http://tutankhamon.ftp.acc.umu.se" target="_blank">tutankhamon.ftp.acc.umu.se</a> (2001:6b0:19::163): icmp_seq=2 ttl=51 time=43.9 ms<br>> 64 bytes from <a href="http://tutankhamon.ftp.acc.umu.se" target="_blank">tutankhamon.ftp.acc.umu.se</a> (2001:6b0:19::163): icmp_seq=3 ttl=51 time=42.7 ms<br>> 64 bytes from <a href="http://tutankhamon.ftp.acc.umu.se" target="_blank">tutankhamon.ftp.acc.umu.se</a> (2001:6b0:19::163): icmp_seq=4 ttl=51 time=49.0 ms<br>> ^C<br>> --- <a href="http://ftp.sunet.se" target="_blank">ftp.sunet.se</a> ping statistics ---<br>> 4 packets transmitted, 4 received, 0% packet loss, time 3005ms<br>> rtt min/avg/max/mdev = 42.708/45.419/49.019/2.404 ms<br>><br>> Speedtest now gives decent speed, 50Mbit/s down and 13.4Mbit/ up where i before only got +1000ms latency and 7Mbit/s down where I am (just outside of central stockholm)<br>> <a href="https://www.speedtest.net/result/11435773983" target="_blank">https://www.speedtest.net/result/11435773983</a><br>><br>> Anything i can do to test / debug interrupts without intel vt-d enabled while I have this fresh in memory?<br><br>Yes, without intel VT-D (and IRQ_REMAP kernel config), it's not possible to request multiple MSI, that's indeed why you get the warning. Normally, each MHI event ring has its own dedicated MSI (to isolate events). But because we noticed vt-d is not necessarily enabled by default, we introduced a fallback to use a single shared MSI. That also why you get multiple irq names (bhi, mhi, mhi....) for the same IRQ index:<br><br>>> CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7<br>>> 143: 0 0 0 0 0 6 0 192 PCI-MSI 4194304-edge bhi, mhi, mhi, mhi, mhi<br><br><div>related patch: <a href="https://git.kernel.org/pub/scm/linux/kernel/git/mani/mhi.git/commit/?h=mhi-next&id=b91c3b30e2267265cd7e67cb3d0c99c48c02b001" target="_blank">https://git.kernel.org/pub/scm/linux/kernel/git/mani/mhi.git/commit/?h=mhi-next&id=b91c3b30e2267265cd7e67cb3d0c99c48c02b001</a></div><br>This shared IRQ mechanism has been validated with Telit FN980m WWAN module.<br><br>In your case, the modem seems to trigger a different interrupt than the shared one... I would suggest checking in the MHI core, especially in mhi_init_irq_setup() for the host side and in mhi_init_dev_ctxt() for the modem side, that misvec index is set to the same value (0).<br><br>If everything is correct, it's possibly a bug on modem/firmware side.<br><br>Regards,<br>Loic</div>
</blockquote></div>