[systemd-devel] Fwd: What could be causing a oneshot unit not to be run when a machine reboots?

[John Ioannidis]

I have the following service file:

[Unit]
Description=D
After=network.target

[Service]
ExecStart=/usr/local/sbin/mktags.py
Type=oneshot
User=root

[Install]
WantedBy=multi-user.target


Occasionally, when the machine reboots, it does not run. Here is the
evidence:

# uptime
05:50:11 up 20 min,  1 user,  load average: 0.14, 0.03, 0.01

The machine has been up for 20 minutes, so it rebooted at 05:30

# systemctl status mktags.service
* mktags.service - D
    Loaded: loaded (/etc/systemd/system/mktags.service; enabled; vendor
preset: enabled)
    Active: inactive (dead)

Weird... no indication of what the last process was... let's check
journalctl:

# journalctl -u mktags.service | tail -4
-- Reboot --
Mar 14 05:25:22 sll9 systemd[1]: Starting D...
Mar 14 05:25:22 sll9 systemd[1]: mktags.service: Succeeded.
Mar 14 05:25:22 sll9 systemd[1]: Finished D.

The last time the service ran was five minutes *before* I rebooted. Other
times the service runs fine. On the same machine, a bit later:

# uptime
05:57:31 up 2 min,  1 user,  load average: 0.31, 0.33, 0.14


# systemctl status mktags.service
* mktags.service - D
    Loaded: loaded (/etc/systemd/system/mktags.service; enabled; vendor
preset: enabled)
    Active: inactive (dead) since Sun 2021-03-14 05:55:14 UTC; 2min 24s ago
   Process: 468 ExecStart=/usr/local/sbin/mktags.py (code=exited,
status=0/SUCCESS)
  Main PID: 468 (code=exited, status=0/SUCCESS)
(more output indicated that the process indeed ran).



How do I even debug something like this? I have added code in mktags.py to
print stuff to stderr, so that journalctl picks it up. When the service
runs, the expected output is there. Where it does not run, it is not there
of course.

I cannot detect any other errors elsewhere; if the service fails to start,
and I reboot, it invariably runs then. Obviously this is not a solution;
these are unattended machines running on GCE!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20210314/0f79116a/attachment.htm>