<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class="gmail_default" style="font-size:small">Hey,</div><div class="gmail_default" style="font-size:small"><br class=""></div><div class="gmail_default" style="font-size:small">To answer your questions below:</div><div class="gmail_default" style="font-size:small"><br class=""></div><div class="gmail_default" style="font-size:small">1. We are not actively resetting the modem anymore. We found that this only made matters worse. Essentially the kernel was not releasing the USB interfaces of the modem completely, which results in ModemManager not acquiring the right ports. </div><div class="gmail_default" style="font-size:small">2. We are now using mmcli to turn on the GPS. This works without any issues.</div><div class="gmail_default" style="font-size:small">3. As you said, it is only possible to have 1 instance of ModemManager, I thought there were 3 separate instances, but this is not the case.</div><div class="gmail_default" style="font-size:small">4. We are not using qmicli ourselves to communicate with the modem via the QMI interface, all interactions are done by ModemManager.</div><div class="gmail_default" style="font-size:small">5. We have updated ModemManager to 1.12.2 and libqmi to 1.24.2</div><div class="gmail_default" style="font-size:small"><br class=""></div><div class="gmail_default" style="font-size:small">The problem we see with ModemManager is that it is slowly using more and more memory while using the QMI interface of our Quectel BG96 modem.</div><div class="gmail_default" style="font-size:small"><div class="gmail_default">We have done some more research and found the following things:</div><div class="gmail_default"><br class=""></div><div class="gmail_default">Our system has 512MB RAM. The OOM killer kills ModemManager after 12 hours because at that time, it is using half of the memory available:</div><div class="gmail_default"><br class=""></div><div class="gmail_default"><i class="">[43036.640048] kworker/u2:1 invoked oom-killer: gfp_mask=0x14082c2(GFP_KERNEL|__GFP_HIGHMEM|__GFP_NOWARN|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=0</i></div><i class="">[43036.697846] kworker/u2:1 cpuset=/ mems_allowed=0<br class="">[43036.701205] CPU: 0 PID: 18834 Comm: kworker/u2:1 Not tainted 4.14.78 #1<br class="">[43036.706521] Hardware name: Freescale i.MX6 UltraLite (Device Tree)<br class="">[43036.711544] Workqueue: brcmf_wq/mmc0:0001:1 brcmf_sdio_dataworker [brcmfmac]<br class="">[43036.717326] [<8010f7b0>] (unwind_backtrace) from [<8010b2bc>] (show_stack+0x10/0x14)<br class="">[43036.723780] [<8010b2bc>] (show_stack) from [<80a2cf5c>] (dump_stack+0x78/0x8c)<br class="">[43036.729713] [<80a2cf5c>] (dump_stack) from [<801ceee8>] (dump_header+0xa0/0x230)<br class="">[43036.735819] [<801ceee8>] (dump_header) from [<801ce1f8>] (oom_kill_process+0xb0/0x514)<br class="">[43036.742444] [<801ce1f8>] (oom_kill_process) from [<801cec2c>] (out_of_memory+0x15c/0x31c)<br class="">[43036.749329] [<801cec2c>] (out_of_memory) from [<801d3884>] (__alloc_pages_nodemask+0xd14/0xf34)<br class="">[43036.756739] [<801d3884>] (__alloc_pages_nodemask) from [<802054a0>] (__vmalloc_node_range+0x114/0x230)<br class="">[43036.764757] [<802054a0>] (__vmalloc_node_range) from [<80205858>] (__vmalloc_node.constprop.11+0x48/0x50)<br class="">[43036.773030] [<80205858>] (__vmalloc_node.constprop.11) from [<802058c0>] (vzalloc+0x2c/0x34)<br class="">[43036.780267] [<802058c0>] (vzalloc) from [<7f0369f0>] (brcmf_sdio_dataworker+0x1cbc/0x216c [brcmfmac])<br class="">[43036.788286] [<7f0369f0>] (brcmf_sdio_dataworker [brcmfmac]) from [<80141298>] (process_one_work+0x1d8/0x410)<br class="">[43036.796825] [<80141298>] (process_one_work) from [<80141f2c>] (worker_thread+0x50/0x598)<br class="">[43036.803623] [<80141f2c>] (worker_thread) from [<80146d90>] (kthread+0x14c/0x154)<br class="">[43036.809728] [<80146d90>] (kthread) from [<80107a48>] (ret_from_fork+0x14/0x2c)<br class="">[43037.167843] Mem-Info:<br class="">[43037.168859] active_anon:98149 inactive_anon:17916 isolated_anon:0<br class=""> active_file:12 inactive_file:25 isolated_file:0<br class=""> unevictable:0 dirty:0 writeback:0 unstable:0<br class=""> slab_reclaimable:1741 slab_unreclaimable:4386<br class=""> mapped:14080 shmem:33723 pagetables:936 bounce:0<br class=""> free:650 free_pcp:45 free_cma:0<br class="">[43037.277821] Node 0 active_anon:392596kB inactive_anon:71664kB active_file:48kB inactive_file:52kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:56320kB dirty:0kB writeback:0kB shmem:134892kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no<br class="">[43037.367828] Normal free:2680kB min:2704kB low:3380kB high:4056kB active_anon:392596kB inactive_anon:71664kB active_file:48kB inactive_file:52kB unevictable:0kB writepending:0kB present:524288kB managed:503600kB mlocked:0kB kernel_stack:1696kB pagetables:3744kB bounce:0kB free_pcp:120kB local_pcp:120kB free_cma:0kB<br class="">[43037.477797] lowmem_reserve[]: 0 0 0<br class="">[43037.480024] Normal: 82*4kB (UME) 72*8kB (UME) 31*16kB (UM) 16*32kB (UM) 6*64kB (U) 3*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB = 2680kB<br class="">[43037.541295] 33751 total pagecache pages<br class="">[43037.543847] 0 pages in swap cache<br class="">[43037.545863] Swap cache stats: add 0, delete 0, find 0/0<br class="">[43037.587944] Free swap = 0kB<br class="">[43037.589582] Total swap = 0kB<br class="">[43037.591169] 131072 pages RAM<br class="">[43037.592750] 0 pages HighMem/MovableOnly<br class="">[43037.595286] 5172 pages reserved<br class="">[43037.597126] 0 pages cma reserved<br class="">[43037.617826] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name<br class="">[43037.657894] [ 109] 0 109 7060 2073 50 0 0 0 systemd-journal<br class="">[43037.666094] [ 115] 0 115 2058 1069 8 0 0 -1000 systemd-udevd<br class="">[43037.713088] [ 132] 1000 132 1695 1139 7 0 0 -900 dbus-daemon<br class="">[43037.743755] [ 133] 0 133 1896 1092 8 0 0 0 haveged<br class="">[43037.787950] [ 135] 0 135 63780 53513 116 0 0 0 ModemManager<br class="">[43037.795893] [ 138] 0 138 1849 856 8 0 0 0 systemd-logind<br class="">[43037.837831] [ 160] 0 160 635 74 5 0 0 0 getty<br class="">[43037.845169] [ 186] 0 186 859 582 6 0 0 0 modem.sh<br class="">[43037.917936] [ 193] 1006 193 1971 845 8 0 0 0 systemd-network<br class="">[43037.926136] [ 196] 0 196 17518 3188 23 0 0 0 NetworkManager<br class="">[43037.971235] [ 244] 1007 244 2608 1599 10 0 0 0 systemd-resolve<br class="">[43038.004486] [ 245] 0 245 1574 1221 6 0 0 0 openvpn<br class="">[43038.042983] [ 275] 0 275 826 600 6 0 0 0 led-control.sh<br class="">[43038.077907] [ 276] 0 276 3115 469 8 0 0 0 chronyd<br class="">[43038.085472] [ 304] 0 304 2073 1085 8 0 0 0 wpa_supplicant<br class="">[43038.126229] [ 320] 0 320 6909 899 12 0 0 0 qmi-proxy<br class="">[43038.177919] [ 366] 0 366 240369 9498 48 0 0 -500 balena-engine-d<br class="">[43038.186169] [ 386] 0 386 238336 7960 45 0 0 -500 balena-engine-c<br class="">[43038.224253] [ 398] 0 398 1350 861 7 0 0 -1000 sshd<br class="">[43038.267853] [ 578] 0 578 225786 6921 35 0 0 -999 balena-engine-c<br class="">[43038.276058] [ 585] 0 585 223993 7224 33 0 0 -999 balena-engine-c<br class="">[43038.333144] [ 587] 0 587 226042 7248 38 0 0 -999 balena-engine-c<br class="">[43038.357927] [ 602] 0 602 223993 7040 34 0 0 -999 balena-engine-c<br class="">[43038.397439] [ 688] 0 688 1290 75 7 0 0 0 enqueue<br class="">[43038.427860] [ 696] 0 696 42418 5093 104 0 0 0 node<br class="">[43038.467867] [ 702] 999 702 4385 206 9 0 0 0 redis-server<br class="">[43038.475915] [ 717] 0 717 32997 6244 106 0 0 0 datahub-stats<br class="">[43038.527834] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout<br class="">[43038.532447] brcmfmac: _brcmf_set_multicast_list: Setting allmulti failed, -110<br class="">[43038.538498] [ 934] 0 934 826 551 6 0 0 0 app-layer-updat<br class="">[43038.538511] [ 935] 0 935 826 588 7 0 0 0 app-layer-healt<br class="">[43038.538520] [11460] 0 11460 1380 185 7 0 0 0 sleep<br class="">[43038.538532] [ 6036] 0 6036 228091 6967 36 0 0 -999 balena-engine-c<br class="">[43038.538542] [ 6063] 0 6063 29799 2932 74 0 0 0 ivh2-msgs-deque<br class="">[43038.538553] [18785] 0 18785 1380 184 7 0 0 0 sleep<br class="">[43038.538563] [18901] 0 18901 213220 4860 24 0 0 0 balena-engine<br class="">[43038.538575] [18902] 0 18902 540 74 4 0 0 0 grep<br class="">[43038.538584] [18907] 0 18907 826 321 5 0 0 0 led-control.sh<br class="">[43038.538639] Out of memory: Kill process 135 (ModemManager) score 413 or sacrifice child<br class="">[43038.538685] Killed process 135 (ModemManager) total-vm:255120kB, anon-rss:204780kB, file-rss:4kB, shmem-rss:9268kB<br class="">[43039.110517] oom_reaper: reaped process 135 (ModemManager), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB</i></div><div class="gmail_default" style="font-size:small"><br class=""></div><div class="gmail_default" style="font-size:small">After we found out that the OOM killer was killing ModemManager we ran ModemManager in debug mode using Memcheck from Valgrind to try to capture the memory leaks. See the log below</div><div class="gmail_default" style="font-size:small"><br class=""></div><div class="gmail_default"><div class="gmail_default"><font size="2" class=""><i class="">==2705== Memcheck, a memory error detector</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== Command: /usr/sbin/ModemManager --debug --filter-policy=WHITELIST-ONLY</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== Parent PID: 2598</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== Warning: noted but unhandled ioctl 0x540c with no size/direction hints.</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== This could cause spurious value errors to appear.</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== HEAP SUMMARY:</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== in use at exit: 237,156 bytes in 5,555 blocks</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== total heap usage: 30,658 allocs, 25,103 frees, 8,699,112 bytes allocated</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 12 bytes in 1 blocks are indirectly lost in loss record 1 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x4847D74: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 32 bytes in 4 blocks are possibly lost in loss record 2 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x4847D74: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 32 bytes in 4 blocks are possibly lost in loss record 3 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x4847C88: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 52 (40 direct, 12 indirect) bytes in 4 blocks are definitely lost in loss record 4 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x4847D74: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 376 bytes in 4 blocks are possibly lost in loss record 5 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x484AA44: realloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 680 bytes in 16 blocks are possibly lost in loss record 6 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x484A7C4: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 688 bytes in 43 blocks are still reachable in loss record 7 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x4DEF9E4: g_closure_new_simple (in /usr/lib/libgobject-2.0.so.0.5600.4)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 928 bytes in 43 blocks are still reachable in loss record 8 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x4DEF99C: g_closure_new_simple (in /usr/lib/libgobject-2.0.so.0.5600.4)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 4,080 bytes in 2 blocks are definitely lost in loss record 9 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x484A7C4: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 8,684 bytes in 296 blocks are still reachable in loss record 10 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x4847C88: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 8,700 bytes in 217 blocks are still reachable in loss record 11 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x4E0AF58: g_type_create_instance (in /usr/lib/libgobject-2.0.so.0.5600.4)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 10,512 bytes in 217 blocks are still reachable in loss record 12 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x4E0AF9C: g_type_create_instance (in /usr/lib/libgobject-2.0.so.0.5600.4)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 12,067 bytes in 116 blocks are still reachable in loss record 13 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x484AA44: realloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 73,835 bytes in 3,023 blocks are still reachable in loss record 14 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x4847D74: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== 93,754 bytes in 1,305 blocks are still reachable in loss record 15 of 15</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== at 0x484A7C4: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== LEAK SUMMARY:</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== definitely lost: 4,120 bytes in 6 blocks</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== indirectly lost: 12 bytes in 1 blocks</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== possibly lost: 1,120 bytes in 28 blocks</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== still reachable: 209,168 bytes in 5,260 blocks</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== of which reachable via heuristic:</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== newarray : 1,976 bytes in 75 blocks</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== suppressed: 0 bytes in 0 blocks</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== For counts of detected and suppressed errors, rerun with: -v</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2756== ERROR SUMMARY: 6 errors from 6 contexts (suppressed: 0 from 0)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== HEAP SUMMARY:</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== in use at exit: 874,364 bytes in 24,674 blocks</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== total heap usage: 2,690,239 allocs, 2,665,565 frees, 812,938,818 bytes allocated</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== 32 bytes in 4 blocks are possibly lost in loss record 1 of 14</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== at 0x4847C88: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== 480 bytes in 4 blocks are possibly lost in loss record 2 of 14</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== at 0x484AA44: realloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== 728 bytes in 10 blocks are possibly lost in loss record 3 of 14</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== at 0x4847D74: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== 1,032 bytes in 18 blocks are possibly lost in loss record 4 of 14</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== at 0x484A7C4: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== 1,744 bytes in 109 blocks are still reachable in loss record 5 of 14</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== at 0x4DEF9E4: g_closure_new_simple (in /usr/lib/libgobject-2.0.so.0.5600.4)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== 2,256 bytes in 109 blocks are still reachable in loss record 6 of 14</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== at 0x4DEF99C: g_closure_new_simple (in /usr/lib/libgobject-2.0.so.0.5600.4)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== 9,244 bytes in 364 blocks are still reachable in loss record 7 of 14</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== at 0x4847C88: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== 15,776 bytes in 470 blocks are still reachable in loss record 8 of 14</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== at 0x4E0AF9C: g_type_create_instance (in /usr/lib/libgobject-2.0.so.0.5600.4)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== 17,152 bytes in 229 blocks are still reachable in loss record 9 of 14</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== at 0x484AA44: realloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== 23,512 bytes in 470 blocks are still reachable in loss record 10 of 14</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== at 0x4E0AF58: g_type_create_instance (in /usr/lib/libgobject-2.0.so.0.5600.4)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== 78,223 bytes in 498 blocks are indirectly lost in loss record 11 of 14</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== at 0x4847D74: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== 93,063 bytes in 3,062 blocks are still reachable in loss record 12 of 14</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== at 0x4847D74: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== 110,326 bytes in 1,844 blocks are still reachable in loss record 13 of 14</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== at 0x484A7C4: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== 551,535 (473,312 direct, 78,223 indirect) bytes in 16,904 blocks are definitely lost in loss record 14 of 14</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== at 0x4847D74: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== LEAK SUMMARY:</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== definitely lost: 473,312 bytes in 16,904 blocks</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== indirectly lost: 78,223 bytes in 498 blocks</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== possibly lost: 2,272 bytes in 36 blocks</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== still reachable: 273,073 bytes in 6,657 blocks</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== of which reachable via heuristic:</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== newarray : 5,276 bytes in 150 blocks</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== suppressed: 0 bytes in 0 blocks</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705==</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== For counts of detected and suppressed errors, rerun with: -v</i></font></div><div class="gmail_default"><font size="2" class=""><i class="">==2705== ERROR SUMMARY: 5 errors from 5 contexts (suppressed: 0 from 0)</i></font></div></div><div class="gmail_default" style="font-size:small"><br class=""></div><div class="gmail_default" style="font-size:small">As you can see we are not able to get the stack trace to work with valgrind. The command we are currently using is:</div><div class="gmail_default" style="font-size:small"><br class=""></div><div class="gmail_default" style="font-size:small"><i class="">systemctl stop ModemManager && G_SLICE=always-malloc valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --log-file=/data/valgrind.out /usr/bin/ModemManager --debug --filter-policy=WHITELIST-ONLY </i></div><div class="gmail_default" style="font-size:small"><br class=""></div><div class="gmail_default" style="font-size:small">Can you help us with getting the stack traces of the memory leaks that memcheck finds?</div><div class="gmail_default" style="font-size:small"><br class=""></div><div class="gmail_default" style="font-size:small">Geert<br class=""></div><div class="gmail_default" style="font-size:small"><br class=""></div></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 25 Nov 2019 at 11:40, Aleksander Morgado <<a href="mailto:aleksander@aleksander.es" class="">aleksander@aleksander.es</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hey,<br class=""> <br class=""> ><br class=""> > We are using ModemManager with the Quectel BG96 modem and are using it's QMI interface.<br class=""> > Everything is going well, expect that ModemManager starts to use a lot of the memory after it is running for a while.<br class=""> ><br class=""> <br class=""> That's unexpected, MM shouldn't start using lots of memory unless<br class=""> there's some obvious memory leak somewhere.</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br class=""> > The Quectel BG96 modem has been resetted 3 times during its operation by sending "AT+CFUN=1,1" to the primary AT interface.<br class=""> > This is done, by a script we have running on the device, when the connection is lost for 5 minutes.<br class=""> <br class=""> How do you do this? Are you talking to the TTY port directly, or are<br class=""> you running CFUN=1,1 through mmcli?<br class=""> Have you tried to run mmcli -m X --reset instead?<br class=""> <br class=""> > After the modem is reset, the script checks if the ModemManager can see the modem and will check the GPS settings of the modem. This is done via the primary AT interface.<br class=""> <br class=""> Again, how? via mmcli? or via the TTY port directly?<br class=""> <br class=""> > After this, we turn on the GPS via ModemManager using the Location interface.<br class=""> ><br class=""> > Commands used for enabling GPS ( $modem_id is requested from DBus ):<br class=""> > mmcli -m $modem_id --location-enable-gps-nmea<br class=""> > mmcli -m $modem_id --location-enable-gps-raw<br class=""> > mmcli -m $modem_id --location-set-enable-signal<br class=""> > mmcli -m $modem_id--location-set-gps-refresh-rate=0<br class=""> ><br class=""> > When this is done, we setup the cellular connection using NetworkManager.<br class=""> ><br class=""> <br class=""> Ok.<br class=""> <br class=""> > Currently we see that this has produced ModemManager running 3 instances of itself and using 25% of the 512MB of memory.<br class=""> <br class=""> This is not truly possible; there can only be one ModemManager daemon<br class=""> running at a time, because only one process can acquire the well-known<br class=""> DBus name in the system bus. I have really never seen more than 1<br class=""> instance of MM running at the same time. Why do you say there are 3?<br class=""> do you really see 3 ModemManager processes with 3 different PIDs all<br class=""> running at the same time?<br class=""> <br class=""> > What also happens is that the GPS is not always turned on after a modem reset.<br class=""> ><br class=""> > Expected state:<br class=""> ><br class=""> > $ mmcli -m 0 --location-status<br class=""> > --------------------------------<br class=""> > Location | capabilities: 3gpp-lac-ci, gps-raw, gps-nmea, agps<br class=""> > | enabled: 3gpp-lac-ci, gps-raw, gps-nmea<br class=""> > | signals: yes<br class=""> > --------------------------------<br class=""> > GPS | refresh rate: 0 seconds<br class=""> ><br class=""> > Actual state:<br class=""> ><br class=""> > $ mmcli -m 0 --location-status<br class=""> > --------------------------------<br class=""> > Location | capabilities: 3gpp-lac-ci, gps-raw, gps-nmea, agps<br class=""> > | enabled: 3gpp-lac-ci<br class=""> > | signals: no<br class=""> > -------------------------------<br class=""> > GPS | refresh rate: 30 seconds<br class=""> <br class=""> This could be because MM couldn't grab the QMI interface for some<br class=""> reason. There is one QMI interface usually, right? We would need to<br class=""> see MM debug logs when this happens to understand why the QMI port<br class=""> probing failed.<br class=""> <br class=""> ><br class=""> > Are we doing something wrong in the way we are resetting the modem after a long duration of connection loss?<br class=""> ><br class=""> <br class=""> Depends on how you talk to the TTY port to run the CFUN command<br class=""> really. If you could try with mmcli --reset, that may also give good<br class=""> results to do so. Although, ideally, there shouldn't be any reason why<br class=""> you need to reset the modem :)<br class=""> <br class=""> > Do we need to restart ModemManager each time we reset the modem to prevent multiple instances of ModemManager to be running at the same time?<br class=""> ><br class=""> <br class=""> Need more information about this issue, because there's no way this<br class=""> can happen really.<br class=""> <br class=""> > What we also see is that ModemManager is giving a lot of warnings, does this has something to do with the problems we are experiencing?<br class=""> ><br class=""> > Logs from ModemManager:<br class=""> ><br class=""> > Nov 25 07:04:39 ModemManager[158]: <warn> Reloading stats failed: QMI operation failed: Transaction timed out<br class=""> > Nov 25 07:04:41 ModemManager[158]: transaction 0x26e aborted, but message is not abortable<br class=""> > Nov 25 07:04:59 ModemManager[158]: [/dev/cdc-wdm0] No transaction matched in received message<br class=""> > Nov 25 07:04:59 ModemManager[158]: [/dev/cdc-wdm0] No transaction matched in received message<br class=""> > Nov 25 07:05:09 ModemManager[158]: transaction 0x26f aborted, but message is not abortable<br class=""> > Nov 25 07:05:09 ModemManager[158]: <warn> Reloading stats failed: QMI operation failed: Transaction timed out<br class=""> > Nov 25 07:06:39 ModemManager[158]: transaction 0x272 aborted, but message is not abortable<br class=""> > Nov 25 07:06:39 ModemManager[158]: <warn> Reloading stats failed: QMI operation failed: Transaction timed out<br class=""> > Nov 25 07:06:41 ModemManager[158]: [/dev/cdc-wdm0] No transaction matched in received message<br class=""> > Nov 25 07:49:28 ModemManager[158]: transaction 0x2c7 aborted, but message is not abortable<br class=""> > Nov 25 08:05:09 ModemManager[158]: transaction 0x2e7 aborted, but message is not abortable<br class=""> > Nov 25 08:05:09 ModemManager[158]: <warn> Reloading stats failed: QMI operation failed: Transaction timed out<br class=""> > Nov 25 08:09:39 ModemManager[158]: transaction 0x2f0 aborted, but message is not abortable<br class=""> > Nov 25 08:09:39 ModemManager[158]: <warn> Reloading stats failed: QMI operation failed: Transaction timed out<br class=""> > Nov 25 08:10:39 ModemManager[158]: transaction 0x2f2 aborted, but message is not abortable<br class=""> > Nov 25 08:10:39 ModemManager[158]: <warn> Reloading stats failed: QMI operation failed: Transaction timed out<br class=""> > Nov 25 08:29:39 ModemManager[158]: transaction 0x318 aborted, but message is not abortable<br class=""> > Nov 25 08:29:39 ModemManager[158]: <warn> Reloading stats failed: QMI operation failed: Transaction timed out<br class=""> > Nov 25 08:45:09 ModemManager[158]: transaction 0x337 aborted, but message is not abortable<br class=""> > Nov 25 08:45:09 ModemManager[158]: <warn> Reloading stats failed: QMI operation failed: Transaction timed out<br class=""> > Nov 25 09:03:39 ModemManager[158]: transaction 0x35c aborted, but message is not abortable<br class=""> > Nov 25 09:03:39 ModemManager[158]: <warn> Reloading stats failed: QMI operation failed: Transaction timed out<br class=""> > Nov 25 09:06:09 ModemManager[158]: transaction 0x361 aborted, but message is not abortable<br class=""> > Nov 25 09:06:09 ModemManager[158]: <warn> Reloading stats failed: QMI operation failed: Transaction timed out<br class=""> ><br class=""> <br class=""> Those timeouts could indicate that the QMI port is not responding. Are<br class=""> you using the QMI port through qmicli yourselves in any way?<br class=""> <br class=""> > Version of ModemManager used: 1.10.8<br class=""> > Version of libqmi used: 1.24.0<br class=""> <br class=""> You should update to MM 1.12! :)<br class=""> <br class=""> -- <br class=""> Aleksander<br class=""> <a href="https://aleksander.es/" rel="noreferrer" target="_blank" class="">https://aleksander.es</a><br class=""> </blockquote></div></div> </body></html>