<html><head></head><body><span style="font-family: monospace;">I build an mostly-stateless appliance OS derived from Fedora (25 ATM)</span><br style="font-family: monospace;"><span style="font-family: monospace;">and have several custom units to make it all possible. My units had</span><br style="font-family: monospace;"><span style="font-family: monospace;">worked great with F21, but are now giving me problems with F25. One</span><br style="font-family: monospace;"><span style="font-family: monospace;">pair of the custom units do some trickery to relocate sshd host keys</span><br style="font-family: monospace;"><span style="font-family: monospace;">from /etc/ssh to an alternate location that provides persistence:</span><br style="font-family: monospace;"><br style="font-family: monospace;"><span style="font-family: monospace;">$ systemctl cat sshd-persist-keys.service</span><br style="font-family: monospace;"><span style="font-family: monospace;"># /usr/lib/systemd/system/sshd-persist-keys.service</span><br style="font-family: monospace;"><span style="font-family: monospace;">[Unit]</span><br style="font-family: monospace;"><span style="font-family: monospace;">Description=OpenSSH server - persist volatile keys for the AOS</span><br style="font-family: monospace;"><span style="font-family: monospace;">After=sshd-keygen.target</span><br style="font-family: monospace;"><span style="font-family: monospace;">Before=sshd.service</span><br style="font-family: monospace;"><span style="font-family: monospace;">Wants=sshd-keygen.target</span><br style="font-family: monospace;"><br style="font-family: monospace;"><span style="font-family: monospace;">[Service]</span><br style="font-family: monospace;"><span style="font-family: monospace;">ExecStart=/usr/sbin/sshd-persist-keys</span><br style="font-family: monospace;"><span style="font-family: monospace;">Type=oneshot</span><br style="font-family: monospace;"><span style="font-family: monospace;">RemainAfterExit=yes</span><br style="font-family: monospace;"><br style="font-family: monospace;"><span style="font-family: monospace;">[Install]</span><br style="font-family: monospace;"><span style="font-family: monospace;">WantedBy=multi-user.target</span><br style="font-family: monospace;"><br style="font-family: monospace;"><br style="font-family: monospace;"><span style="font-family: monospace;">$ systemctl cat sshd-restore-keys.service</span><br style="font-family: monospace;"><span style="font-family: monospace;"># /usr/lib/systemd/system/sshd-restore-keys.service</span><br style="font-family: monospace;"><span style="font-family: monospace;">[Unit]</span><br style="font-family: monospace;"><span style="font-family: monospace;">Description=OpenSSH server - restore persisted keys for the AOS</span><br style="font-family: monospace;"><span style="font-family: monospace;">After=aos-storage-init.service</span><br style="font-family: monospace;"><a href="mailto:Before=sshd-keygen@rsa.service" style="font-family: monospace;">Before=sshd-keygen@rsa.service</a><span style="font-family: monospace;"> </span><a href="mailto:sshd-keygen@ecdsa.service" style="font-family: monospace;">sshd-keygen@ecdsa.service</a><span style="font-family: monospace;"> sshd-keygen@ed</span><br style="font-family: monospace;"><span style="font-family: monospace;">25519.service</span><br style="font-family: monospace;"><br style="font-family: monospace;"><span style="font-family: monospace;">[Service]</span><br style="font-family: monospace;"><span style="font-family: monospace;">ExecStart=/usr/sbin/sshd-restore-keys</span><br style="font-family: monospace;"><span style="font-family: monospace;">Type=oneshot</span><br style="font-family: monospace;"><span style="font-family: monospace;">RemainAfterExit=yes</span><br style="font-family: monospace;"><br style="font-family: monospace;"><span style="font-family: monospace;">[Install]</span><br style="font-family: monospace;"><span style="font-family: monospace;">WantedBy=multi-user.target</span><br style="font-family: monospace;"><br style="font-family: monospace;"><br style="font-family: monospace;"><span style="font-family: monospace;">I found that on some boots, sshd wasn't getting started. With the help</span><br style="font-family: monospace;"><span style="font-family: monospace;">of booting with systemd.log_level=debug I learned:</span><br style="font-family: monospace;"><br style="font-family: monospace;"><span style="font-family: monospace;">$ sudo journalctl | grep conflict</span><br style="font-family: monospace;"><span style="font-family: monospace;">Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job</span><br style="font-family: monospace;"><span style="font-family: monospace;">sshd.service/start conflicted_by=no</span><br style="font-family: monospace;"><span style="font-family: monospace;">Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job</span><br style="font-family: monospace;"><span style="font-family: monospace;">sshd.service/stop conflicted_by=yes</span><br style="font-family: monospace;"><span style="font-family: monospace;">Mar 22 16:11:42 localhost systemd[1]: sshd.service: Fixing conflicting</span><br style="font-family: monospace;"><span style="font-family: monospace;">jobs sshd.service/start,sshd.service/stop by deleting job</span><br style="font-family: monospace;"><span style="font-family: monospace;">sshd.service/start</span><br style="font-family: monospace;"><span style="font-family: monospace;">Mar 22 16:11:42 localhost systemd[1]: sshd.socket: Looking at job</span><br style="font-family: monospace;"><span style="font-family: monospace;">sshd.socket/stop conflicted_by=no</span><br style="font-family: monospace;"><span style="font-family: monospace;">Mar 22 16:11:42 localhost systemd[1]: sshd.socket: Looking at job</span><br style="font-family: monospace;"><span style="font-family: monospace;">sshd.socket/start conflicted_by=no</span><br style="font-family: monospace;"><span style="font-family: monospace;">Mar 22 16:11:42 localhost systemd[1]: sshd.socket: Fixing conflicting</span><br style="font-family: monospace;"><span style="font-family: monospace;">jobs sshd.socket/stop,sshd.socket/start by deleting job</span><br style="font-family: monospace;"><span style="font-family: monospace;">sshd.socket/stop</span><br style="font-family: monospace;"><span style="font-family: monospace;">Mar 22 16:11:42 localhost systemd[1]: sshd-keygen.target: Looking at</span><br style="font-family: monospace;"><span style="font-family: monospace;">job sshd-keygen.target/start conflicted_by=no</span><br style="font-family: monospace;"><span style="font-family: monospace;">Mar 22 16:11:42 localhost systemd[1]: sshd-keygen.target: Looking at</span><br style="font-family: monospace;"><span style="font-family: monospace;">job sshd-keygen.target/stop conflicted_by=no</span><br style="font-family: monospace;"><span style="font-family: monospace;">Mar 22 16:11:42 localhost systemd[1]: sshd-keygen.target: Fixing</span><br style="font-family: monospace;"><span style="font-family: monospace;">conflicting jobs sshd-keygen.target/start,sshd-keygen.target/stop by</span><br style="font-family: monospace;"><span style="font-family: monospace;">deleting job sshd-keygen.target/stop</span><br style="font-family: monospace;"><br style="font-family: monospace;"><br style="font-family: monospace;"><span style="font-family: monospace;">I'm sure systemd-analyze is my friend here, but I've not found an</span><br style="font-family: monospace;"><span style="font-family: monospace;">effective way to grok the problem. s-a dot either gives me too much to</span><br style="font-family: monospace;"><span style="font-family: monospace;">comprehend or not enough to make evident the problem. I've tried</span><br style="font-family: monospace;"><span style="font-family: monospace;">feeding it different PATTERNS like: none at all, sshd.service, ssh-</span><br style="font-family: monospace;"><span style="font-family: monospace;">keygen.target, sshd-{persist,restore}-keys.service and more. I either</span><br style="font-family: monospace;"><span style="font-family: monospace;">see a whole forest or just a few trees that seem to be getting along</span><br style="font-family: monospace;"><span style="font-family: monospace;">happily together.</span><br style="font-family: monospace;"><br style="font-family: monospace;"><span style="font-family: monospace;">What advice can you suggest?</span></body></html>