<div dir="ltr"><div>tl;dr: a .path unit does not appear to be waiting for the After= unit to run first.</div><div><br></div>I am still trying to understand why some services occasionally do not start at boot time. It is a very intermittent behavior, but I caught another instance. Everything is running in Google Compute Engine or Amazon EC2. <div><br></div><div>I have a one-shot service, <b>mktags.service</b>, a long-running job of type <b>notify</b>, <b>ccistated.service</b>, and a path unit with its corresponding service, <b>workdir.path</b> and <b>workdir.service</b>. Here are the relevant parts of the unit files:</div><div><br></div><div><span style="font-family:monospace"><b><span style="color:rgb(0,0,0)">mktags.service</span><br></b></span></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">[Unit] </span></span></div><div><span style="font-family:monospace">Description=Populate /run/metadata/tags </span></div><div><span style="font-family:monospace">After=network.target </span></div><div><span style="font-family:monospace">[Service] </span></div><div><span style="font-family:monospace">ExecStart=/usr/local/sbin/mktags.py </span></div><div><span style="font-family:monospace">Type=oneshot </span></div><div><span style="font-family:monospace">[Install] </span></div><div><span style="font-family:monospace">WantedBy=multi-user.target</span></div></blockquote><div><span style="font-family:monospace"><br></span></div><div><b><span style="color:rgb(0,0,0);font-family:monospace">ccistated.service</span><br></b></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">[Unit] </span></span></div><div><span style="font-family:monospace">Description=State Machine Manager </span></div><div><span style="font-family:monospace">After=mktags.service </span></div><div><span style="font-family:monospace">ConditionPathExists=/run/metadata/tags/resource_class </span></div><div><span style="font-family:monospace">[Service] </span></div><div><span style="font-family:monospace">ExecStart=/usr/local/sbin/ccistated.py </span></div><div><span style="font-family:monospace">Type=notify </span></div><div><span style="font-family:monospace">NotifyAccess=all </span></div><div><span style="font-family:monospace">[Install] </span></div><div><span style="font-family:monospace">WantedBy = multi-user.target</span></div></blockquote><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)"><br></span></span></div><div><span style="font-family:monospace"><b><span style="color:rgb(0,0,0)">workdir.path </span><br></b></span></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><span style="font-family:monospace">[Unit] </span></div><div><span style="font-family:monospace">Description=Trigger workdir.service when a job starts, creating a directory in /opt/circleci/workdir </span></div><div><span style="font-family:monospace;background-color:rgb(0,255,255)">After=ccistated.service </span></div><div><span style="font-family:monospace">ConditionPathExists=/run/metadata/tags/resource_class </span></div><div><span style="font-family:monospace">[Path]</span><br></div><div><span style="font-family:monospace">PathChanged=/opt/circleci/workdir </span></div><div><span style="font-family:monospace">[Install]</span><br></div><div><span style="font-family:monospace">WantedBy=multi-user.target</span></div></blockquote><div><br></div><div>What <b>mktags</b> does is extract the metadata tags from the metadata service, and populate /run/metadata/tags/. </div><div><br></div><div><span style="color:rgb(0,0,0);font-family:monospace">$ systemctl status mktags.service</span><br></div><div><span style="font-family:monospace">* mktags.service - Populate /run/metadata/tags <br> Loaded: loaded (/etc/systemd/system/mktags.service; enabled; vendor preset: enabled) <br> Active: inactive (dead) since <font color="#000000" style="background-color:rgb(255,255,0)">Sat 2021-03-27 05:34:00 UTC; 10min ago </font><br> Process: 454 ExecStart=/usr/local/sbin/mktags.py (code=exited, status=0/SUCCESS) <br> Main PID: 454 (code=exited, status=0/SUCCESS)<br></span></div><br>Sure enough, it ran:<div><br><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$ cat /run/metadata/tags/resource_class </span><br>waw<br></span></div><div><span style="font-family:monospace"><br></span></div>As did the next service:</div><div><br><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$ systemctl status ccistated.service </span><br><span style="font-weight:bold;color:rgb(84,255,84)">*</span><span style="color:rgb(0,0,0)"> ccistated.service - State Machine Manager </span><br> Loaded: loaded (/etc/systemd/system/ccistated.service; enabled; vendor preset: enabled) <br> Active: <span style="font-weight:bold;color:rgb(84,255,84)">active (running)</span><span style="color:rgb(0,0,0)"> since <span style="background-color:rgb(255,255,0)">Sat 2021-03-27 05:36:00 UTC; 8min ago </span></span><br> Main PID: 1420 (python3) <br> Tasks: 1 (limit: 9544) <br> Memory: 8.5M <br> CGroup: /system.slice/ccistated.service <br> `-1420 python3 /usr/local/sbin/ccistated.py<br></span></div><div><span style="font-family:monospace"><br></span></div>It's OK that it took a couple of minutes for ccistated to start up, it does a few things before sending out the Notify.<div><br></div></div><div>But look at the old workdir.path:</div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$ systemctl status workdir.path </span><br>* workdir.path - Trigger workdir.service when a job starts, creating a directory in /opt/circleci/workdir <br> Loaded: loaded (/etc/systemd/system/workdir.path; enabled; vendor preset: enabled) <br> Active: inactive (dead) <br> Triggers: <span style="font-weight:bold;color:rgb(255,84,84)">*</span><span style="color:rgb(0,0,0)"> workdir.service </span><br> Condition: start <span style="font-weight:bold;color:rgb(215,215,95)">condition failed</span><span style="color:rgb(0,0,0)"> at Sat 2021-03-27 05:33:59 UTC; 22min ago </span><br> `- ConditionPathExists=/run/metadata/tags/resource_class was not met <br><br></span></div><div><div>Huh?!?!?! It's supposed to run after ccistated, and of course after mktags (highlighted in cyan above). It appears to be running anyway, and of course mktags has not gotten a chance to finish, and the ConditionExists file has not been created yet.</div><div><br></div><div>Do .path units not obey the same startup rules?</div><div></div></div></div>