[systemd-devel] Runtime symlinks replaced by hard files, systemctl enable no longer works

[Joey Dodson]

Hello list!



Thanks in advance for any help or thoughts!



*Background:*



I have a system where many ‘/etc/systemd/system’ symlink paths got replaced
by files. Therefore, ‘systemctl enable’ no longer works for a majority of
services. Though somehow there are some services that appear to not have
been affected. This happened for everything
in  ‘/etc/systemd/system/multi-user.target.wants’ and it seems all/many of
the ‘*.target.wants’ directories.



If I delete the hard files, then I can use ‘systemctl enable’ for that
service, but it’s not so simple. Many services have dependencies and
aliases and it will throw the following error message:



‘Failed to execute operation: Invalid argument’



And it may create the symlink for the main unit file, but the dependencies
and aliases have still failed. Therefore it’s going to leave the system in
a broken state unless I can get everything.



*Further context:*


Last time the server was rebooted, it was unable to boot into any kernel,
including our rescue kernel. Only emergency mode was possible through
editing the grub boot options. Somehow the default target was no longer set
to ‘multi-user.target’, but we have no logical explanation for how that
could have happened.





*Questions:*

   1. Has anyone seen something like this before? How could this have
   happened? We suspected rsync/scp from another machine, but since there are
   some symlinks unaffected I think that’s less likely. Is there any mechanism
   of systemd that could have caused it?
   2. Systemd has commands to list where the original unit files are
   located, but I need the opposite. Is it possible to list the paths where
   symlinks will go when using ‘systemctl enable’? The error message above is
   not descriptive and according to search results, it could indicate that a
   service is masked, a unit file has extra spaces, symlinks can’t be created
   or any number of things. Without knowing the path for symlinks, I don't
   know which files to delete to get it working again.
   3. I saw there is a ‘-force’ option for ‘systemctl enable’, but it seems
   that it will only overwrite symlinks, not hard files. Is there another way
   to overwrite files in the intended symlink path?
   4. Is there anything I’m missing here? I’ve never seen this before today
   and am pretty stuck with how this could have happened and how to resolve it
   in a thorough/holistic way.



As before, thanks in advance! I could use all the help I can get.



Regards,

Joey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20240604/3db8be19/attachment-0001.htm>