[systemd-devel] Runtime symlinks replaced by hard files, systemctl enable no longer works

[Cristian Rodríguez]

On Tue, Jun 4, 2024 at 5:03 PM Joey Dodson <jdodsonelements at gmail.com> wrote:
>
> Hello list!
>
>
>
> Thanks in advance for any help or thoughts!
>
>
>
> Background:
>
>
>
> I have a system where many ‘/etc/systemd/system’ symlink paths got replaced by files. Therefore, ‘systemctl enable’ no longer works for a majority of services. Though somehow there are some services that appear to not have been affected. This happened for everything in  ‘/etc/systemd/system/multi-user.target.wants’ and it seems all/many of the ‘*.target.wants’ directories.
>
>
>
> If I delete the hard files, then I can use ‘systemctl enable’ for that service, but it’s not so simple. Many services have dependencies and aliases and it will throw the following error message:
>
>
>
> ‘Failed to execute operation: Invalid argument’
>
>
>
> And it may create the symlink for the main unit file, but the dependencies and aliases have still failed. Therefore it’s going to leave the system in a broken state unless I can get everything.
>
>
>
> Further context:
>
>
> Last time the server was rebooted, it was unable to boot into any kernel, including our rescue kernel. Only emergency mode was possible through editing the grub boot options. Somehow the default target was no longer set to ‘multi-user.target’, but we have no logical explanation for how that could have happened.
>
>
>
>
>
> Questions:
>
> Has anyone seen something like this before? How could this have happened? We suspected rsync/scp from another machine, but since there are some symlinks unaffected I think that’s less likely. Is there any mechanism of systemd that could have caused it?
> Systemd has commands to list where the original unit files are located, but I need the opposite. Is it possible to list the paths where symlinks will go when using ‘systemctl enable’? The error message above is not descriptive and according to search results, it could indicate that a service is masked, a unit file has extra spaces, symlinks can’t be created or any number of things. Without knowing the path for symlinks, I don't know which files to delete to get it working again.
> I saw there is a ‘-force’ option for ‘systemctl enable’, but it seems that it will only overwrite symlinks, not hard files. Is there another way to overwrite files in the intended symlink path?
> Is there anything I’m missing here? I’ve never seen this before today and am pretty stuck with how this could have happened and how to resolve it in a thorough/holistic way.
>


Well..the symlinks where restored from an archive that did not support
symlinks?   assuming you want all those services enabled again you
could write a shell script that replaces hard files to symlinks..
It will happen again if you do not determine where that came from
though. whatever did that needs a blowtorch.