<div dir="ltr"><div class="gmail_quote"><div dir="ltr" class="gmail_attr"><br></div><div dir="ltr">I have the following service file:<div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><span style="font-family:monospace"><span style="color:rgb(178,24,178);background-color:rgb(84,255,255)">[</span><span style="color:rgb(178,24,178)">Unit</span><span style="color:rgb(178,24,178);background-color:rgb(84,255,255)">]</span></span></div><div><span style="font-family:monospace"><span style="color:rgb(24,178,24)">Description</span><span style="color:rgb(0,0,0)">=</span><span style="color:rgb(178,24,24)">D</span></span></div><div><span style="font-family:monospace"><span style="color:rgb(24,178,24)">After</span><span style="color:rgb(0,0,0)">=</span><span style="color:rgb(178,24,24)">network.target</span></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace"><span style="color:rgb(178,24,178)">[Service]</span></span></div><div><span style="font-family:monospace"><span style="color:rgb(24,178,24)">ExecStart</span><span style="color:rgb(0,0,0)">=</span><span style="color:rgb(178,24,24)">/usr/local/sbin/mktags.py</span></span></div><div><span style="font-family:monospace"><span style="color:rgb(24,178,24)">Type</span><span style="color:rgb(0,0,0)">=</span><span style="color:rgb(178,24,24)">oneshot</span></span></div><div><span style="font-family:monospace"><span style="color:rgb(24,178,24)">User</span><span style="color:rgb(0,0,0)">=</span><span style="color:rgb(178,24,24)">root</span></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace"><span style="color:rgb(178,24,178)">[Install]</span></span></div><div><span style="font-family:monospace"><span style="color:rgb(24,178,24)">WantedBy</span><span style="color:rgb(0,0,0)">=</span><span style="color:rgb(178,24,24)">multi-user.target</span></span></div></blockquote><font color="#b21818" face="monospace"><br></font>Occasionally, when the machine reboots, it does not run. Here is the evidence:</div><div dir="ltr"><br></div><div dir="ltr"><span style="font-family:monospace"><span style="color:rgb(0,0,0)"># uptime </span><br> 05:50:11 up 20 min, 1 user, load average: 0.14, 0.03, 0.01<br></span></div><div dir="ltr"><br></div><div dir="ltr">The machine has been up for 20 minutes, so it rebooted at 05:30<br></div><div dir="ltr"><br></div><div dir="ltr"><span style="font-family:monospace"><span style="color:rgb(0,0,0)"># systemctl status mktags.service</span></span></div><div dir="ltr"><span style="font-family:monospace">* mktags.service - D <br> Loaded: loaded (/etc/systemd/system/mktags.service; enabled; vendor preset: enabled)</span></div><div dir="ltr"><span style="font-family:monospace"> Active: inactive (dead)<br></span></div><br>Weird... no indication of what the last process was... let's check journalctl:</div><div class="gmail_quote"><br><div dir="ltr"><span style="font-family:monospace"><span style="color:rgb(0,0,0)"># journalctl -u mktags.service | tail -4</span></span></div><div dir="ltr"><span style="font-family:monospace">-- Reboot --</span></div><div dir="ltr"><span style="font-family:monospace">Mar 14 05:25:22 sll9 systemd[1]: Starting D...</span></div><div dir="ltr"><span style="font-family:monospace">Mar 14 05:25:22 sll9 systemd[1]: mktags.service: Succeeded.</span></div><div dir="ltr"><span style="font-family:monospace">Mar 14 05:25:22 sll9 systemd[1]: Finished D.<br></span></div><div dir="ltr"><span style="font-family:monospace"><br></span></div>The last time the service ran was five minutes <b>before</b> I rebooted. Other times the service runs fine. On the same machine, a bit later:</div><div class="gmail_quote"><br></div><div class="gmail_quote"><span style="font-family:monospace"><span style="color:rgb(0,0,0)"># uptime </span><br> 05:57:31 up 2 min, 1 user, load average: 0.31, 0.33, 0.14 <br><span style="color:rgb(0,0,0)"><br></span></span></div><div class="gmail_quote"><span style="font-family:monospace"><span style="color:rgb(0,0,0)"><br></span></span></div><div class="gmail_quote"><span style="font-family:monospace"><span style="color:rgb(0,0,0)"># systemctl status mktags.service </span><br>* mktags.service - D <br> Loaded: loaded (/etc/systemd/system/mktags.service; enabled; vendor preset: enabled) <br> Active: inactive (dead) since Sun 2021-03-14 05:55:14 UTC; 2min 24s ago <br> Process: 468 ExecStart=/usr/local/sbin/mktags.py (code=exited, status=0/SUCCESS) <br> Main PID: 468 (code=exited, status=0/SUCCESS) <br> (more output indicated that the process indeed ran).</span></div><div class="gmail_quote"><span style="font-family:monospace"><br></span></div><div class="gmail_quote"><span style="font-family:monospace"><br></span></div><div class="gmail_quote"><span style="font-family:monospace"><br></span></div>How do I even debug something like this? I have added code in mktags.py to print stuff to stderr, so that journalctl picks it up. When the service runs, the expected output is there. Where it does not run, it is not there of course.<div><br></div><div>I cannot detect any other errors elsewhere; if the service fails to start, and I reboot, it invariably runs then. Obviously this is not a solution; these are unattended machines running on GCE!</div></div>