<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Matthew,<div class=""> </div><div class="">Thanks for chiming in. I hadn’t found your thread before. Would you mind sharing that dispatcher script? Is that working out satisfactory as a workaround?</div><div class=""> </div><div class="">Best regards,</div><div class="">Sven <div> <blockquote type="cite" class=""><div class="">On 22 Oct 2018, at 16:52, Matthew Starr <<a href="mailto:mstarr@hedonline.com" class="">mstarr@hedonline.com</a>> wrote:</div> <div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Aleksander,<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">This is the exact same issue I reported a couple months ago:<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="https://lists.freedesktop.org/archives/modemmanager-devel/2018-August/006593.html" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://lists.freedesktop.org/archives/modemmanager-devel/2018-August/006593.html</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">I am currently working around it by adding a NetworkManager dispatcher script that on ppp0 going down, I restart ModemManager.<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">I never did find a solution to the issue even after trying some modifications to the serial force close code (src/mm-port-serial.c) in ModemManager. It looks like the force_closed boolean gets set and is never reset, which causes the serial port to never retry communications over the serial port again.<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Regards,<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Matthew Starr<br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Date: Mon, 22 Oct 2018 11:39:37 +0200 From: Sven Schwermer <<a href="mailto:sven@svenschwermer.de" class="">sven@svenschwermer.de</a>> To: Aleksander Morgado <<a href="mailto:aleksander@aleksander.es" class="">aleksander@aleksander.es</a>> Cc: <a href="mailto:modemmanager-devel@lists.freedesktop.org" class="">modemmanager-devel@lists.freedesktop.org</a> Subject: Re: Infinite connection loop Message-ID: <41689DBE-1755-4F85-BFA7- <a href="mailto:97CF6E588F41@svenschwermer.de" class="">97CF6E588F41@svenschwermer.de</a>> Content-Type: text/plain; charset="utf-8" I looked a little closer into this now. Here’s what I think is happening: 1. My cell operator terminates the LCP after some time of inactivity. This is confirmed by my operator. 2. pppd receives the termination request, acknowledges it and the modem hangs up the connection. 3. ModemManager fails to re-acquire serial port lock because of the hangup. 4. ModemManager sets the connection state to ‘registered’. 5. ModemManager force-closes the serial port. 6. NetworkManager registers the lost connection and tries to re-establish it. 7. The ModemManager fails to re-open serial port because it had been force- closed. pppd doesn’t appear to be re-launched. How is this all supposed to work? I’m sure the software stack should be able to handle a terminated connection and re-dial it? Please find attached the debug-level log. The LCP termination happens at Oct 21 14:53:51. Best regards, Sven -------------- next part -------------- A non-text attachment was scrubbed... Name: excerpt.long.gz Type: application/x-gzip Size: 53662 bytes Desc: not available URL: <<a href="https://lists.freedesktop.org/archives/modemmanager-" class="">https://lists.freedesktop.org/archives/modemmanager-</a> devel/attachments/20181022/1e9e348d/attachment.bin> -------------- next part -------------- <blockquote type="cite" class="">On 20 Oct 2018, at 16:26, Sven Schwermer <<a href="mailto:sven@svenschwermer.de" class="">sven@svenschwermer.de</a>> </blockquote>wrote: <blockquote type="cite" class=""> Hi again, I now managed to capture the event in the journal. I limited the auto </blockquote>connect retried to 4 in order to keep the log size manageable. I didn’t have debug-level logging active for NetworkManager at the time. I’ll enable that now and try to reproduce. Please find the log attached. The error seems to be happening at Oct 20 07:44:26. <blockquote type="cite" class=""> Best regards, Sven <ed6e56cf2ac5b6bdb2e3ca97276ebd6d.journal.log> <blockquote type="cite" class="">On 20 Oct 2018, at 13:47, Aleksander Morgado </blockquote></blockquote><<a href="mailto:aleksander@aleksander.es" class="">aleksander@aleksander.es</a>> wrote: <blockquote type="cite" class=""><blockquote type="cite" class=""> Hey, <blockquote type="cite" class="">I am running an embedded system with a u-blox SARA-U270 modem and </blockquote></blockquote></blockquote>NetworkManager 1.12.2 / ModemManager 1.8.0. For some reason, the modem connection seems to break over time. I am not quite sure why that’s the case, but when I look into the journal, I see NetworkManager and ModemManager spewing out log messages at a very high rate. This actually causes significant system load (mostly by dbus-daemon). Here’s an excerpt of the messages: <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class=""> Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect started... Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect state </blockquote></blockquote></blockquote>(4/8): Wait to get fully enabled <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect state </blockquote></blockquote></blockquote>(5/8): Register <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect state </blockquote></blockquote></blockquote>(6/8): Bearer <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect started... Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect state </blockquote></blockquote></blockquote>(4/8): Wait to get fully enabled <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect state </blockquote></blockquote></blockquote>(5/8): Register <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect state </blockquote></blockquote></blockquote>(6/8): Bearer <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <warn> </blockquote></blockquote></blockquote>[1539780869.7776] modem-broadband[ttyACM0]: failed to connect modem: Could not open serial device ttyACM0: it has been forced close <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <info> </blockquote></blockquote></blockquote>[1539780869.7778] device (ttyACM0): state change: prepare -> failed (reason 'unknown', sys-iface-state: 'managed') <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <info> </blockquote></blockquote></blockquote>[1539780869.7807] manager: NetworkManager state is now CONNECTED_LOCAL <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <warn> </blockquote></blockquote></blockquote>[1539780869.7840] device (ttyACM0): Activation: failed for connection 'gprsconnection' <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <info> </blockquote></blockquote></blockquote>[1539780869.7860] device (ttyACM0): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed') <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <info> </blockquote></blockquote></blockquote>[1539780869.7923] policy: auto-activating connection 'gprsconnection' (14e02637-0ffb-3a28-806f-ad60904afe7f) <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <info> </blockquote></blockquote></blockquote>[1539780869.8005] device (ttyACM0): Activation: starting connection 'gprsconnection' (14e02637-0ffb-3a28-806f-ad60904afe7f) <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <info> </blockquote></blockquote></blockquote>[1539780869.8015] device (ttyACM0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed') <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <info> </blockquote></blockquote></blockquote>[1539780869.8043] manager: NetworkManager state is now CONNECTING <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect started... Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect state </blockquote></blockquote></blockquote>(4/8): Wait to get fully enabled <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect state </blockquote></blockquote></blockquote>(5/8): Register <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect state </blockquote></blockquote></blockquote>(6/8): Bearer <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect started... Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect state </blockquote></blockquote></blockquote>(4/8): Wait to get fully enabled <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect state </blockquote></blockquote></blockquote>(5/8): Register <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 ModemManager[1385]: Simple connect state </blockquote></blockquote></blockquote>(6/8): Bearer <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <warn> </blockquote></blockquote></blockquote>[1539780869.9321] modem-broadband[ttyACM0]: failed to connect modem: Could not open serial device ttyACM0: it has been forced close <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <info> </blockquote></blockquote></blockquote>[1539780869.9323] device (ttyACM0): state change: prepare -> failed (reason 'unknown', sys-iface-state: 'managed') <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <info> </blockquote></blockquote></blockquote>[1539780869.9350] manager: NetworkManager state is now CONNECTED_LOCAL <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <warn> </blockquote></blockquote></blockquote>[1539780869.9399] device (ttyACM0): Activation: failed for connection 'gprsconnection' <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <info> </blockquote></blockquote></blockquote>[1539780869.9430] device (ttyACM0): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed') <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <info> </blockquote></blockquote></blockquote>[1539780869.9490] policy: auto-activating connection 'gprsconnection' (14e02637-0ffb-3a28-806f-ad60904afe7f) <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <info> </blockquote></blockquote></blockquote>[1539780869.9569] device (ttyACM0): Activation: starting connection 'gprsconnection' (14e02637-0ffb-3a28-806f-ad60904afe7f) <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <info> </blockquote></blockquote></blockquote>[1539780869.9585] device (ttyACM0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed') <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Oct 17 12:54:29 cf352c1 NetworkManager[677]: <info> </blockquote></blockquote></blockquote>[1539780869.9653] manager: NetworkManager state is now CONNECTING <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class=""> It continues like that forever. My modem network connection config is </blockquote></blockquote></blockquote>as follows: <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class=""> [connection] id=gprsconnection type=gsm autoconnect=true autoconnect-retries=0 [gsm] number=*99# [ipv4] dns-search= method=auto may-fail=false I have been looking through the code and what I don’t understand is </blockquote></blockquote></blockquote>why the ttyACM0 would ever be forcefully closed. What could be an explanation for this? <blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class=""> Any insight is appreciated. </blockquote> Do you have a full log of when this issue happens? If you can reproduce this more or less frequently, could you setup both ModemManager and NetworkManager to dump debug logs and try to get </blockquote></blockquote>it <blockquote type="cite" class=""><blockquote type="cite" class="">reproduced? This issue could be to a wrong synchronization between MM and NM/pppd, in particular related to the CLOCAL settings in each case (i.e. MM shouldn't get HUPs in the serial port unless for the cases where the port is going away). -- Aleksander <a href="https://aleksander.es" class="">https://aleksander.es</a></blockquote></blockquote></blockquote></div></blockquote></div> </div></body></html>