[systemd-devel] after systemd upgrade, cryptdisks won't start except with systemctl start cryptdisks.service but that doesn't work if boot goes to failsafe
Marc MERLIN
marc at merlins.org
Mon Oct 28 16:58:29 UTC 2024
While typing up 2 important details I had forgotten to mention, I
figured out the clue: the cryptdisks initscript never actually ran
before either, and the whole thing just worked thanks to
systemd-cryptsetup that got removed during the upgrade (not sure why)
The 2 clues I was writing:
a) now I'm wondering if before debian stable to debian testing upgrade I
did, cryptsetup commands of /etc/crypttab were happening elsewhere in
systemd and that /etc/init.d/cryptdisks was never actually run
2) after the upgrade, while debugging, I had these:
sauron:/etc/systemd/system# l /lib/systemd/system/cryptdisks-early.service /lib/systemd/system/cryptdisks.service
lrwxrwxrwx 1 root root 9 Oct 18 07:47 /lib/systemd/system/cryptdisks-early.service -> /dev/null
lrwxrwxrwx 1 root root 9 Oct 18 07:47 /lib/systemd/system/cryptdisks.service -> /dev/null
masking the service, and I deleted those 2 files, but now I see that
they were there all along in an older backup, so it looks like
/etc/init.d/cryptdisks was apparently never run as intended but somehow
my disks were decrypted anyway, so I never noticed.
>From there, I finally realized that there is a seperate
systemd-cryptsetup package that somehow
was missing from my system after the upgrade. No idea why :-/
Obviously, things work better with it, and this list is probably not the
right place to figure out why the debian apt install upgrade dropped
that package somehow (it was very big upgrade).
But, this is still a bit of black magic/black box to me, and I would at
least love to learn from all this:
1) why did the original /etc/init.d/cryptdisks initscript work fine
after boot, but refused to ever run (probably dependency hang, but I got
not useful message or debug about it) when I tried to run it from the
incomplete boot rescue shell.
2) why I failed to make my own /lib/systemd/system/cryptdisks.service
that did anything (pasted in original message below)
3) why systemd had to make cryptdisks special and take it over with some
internal binary (probably some edge case reason)?
Thanks,
Marc
On Mon, Oct 28, 2024 at 12:19:10AM -0700, Marc MERLIN wrote:
> This is on debian testing (trix)
> ii cryptsetup 2:2.7.5-1 amd64 disk encryption support - startup scripts
> ii systemd 256.7-2 amd64 system and service manager
>
> So I see that for some reason systemd still uses an initscript, latest version in debian testing, and it seems
> like the deps are wrong, so nothing starts it. The boot also hangs and fails to succeed due to filesytems in fstab
> waiting to be decyrpted
> sauron:~# dpkg -L cryptsetup | grep /etc
> /etc
> /etc/default
> /etc/default/cryptdisks
> /etc/init.d
> /etc/init.d/cryptdisks
> /etc/init.d/cryptdisks-early
>
> systemd-sysv-generator[2541]: SysV service '/etc/init.d/cryptdisks' lacks a native systemd unit file. ~ Automatically generating a unit file for compatibility. Please update package to include a native systemd uni>
> systemd-sysv-generator[2541]: SysV service '/etc/init.d/cryptdisks-early' lacks a native systemd unit file. ~ Automatically generating a unit file for compatibility. Please update package to include a native syste>
> systemd-sysv-generator[2541]: Overwriting existing symlink /run/systemd/generator.late/cryptdisks.service with real service.
>
> It is not started by anything at boot, although it used to, but no idea how
>
> Now, I need to start it manually at boot with
> sauron:~# systemctl start cryptdisks.service
> sauron:~# dmsetup ls
> cryptroot (253:0)
> eswap (253:1)
> pool1 (253:2)
> pool2 (253:3)
> pool3 (253:4)
> pool4 (253:5)
>
> but this only works if the boot succeeds to completion.
> And for that to happy, I need to comment out all the filesystems in /etc/fstab
> that rely on decrypted filetsystems
>
> for instance:
> #LABEL=btrfs_pool1 /mnt/btrfs_pool1 btrfs defaults,nofail,compress=lzo,skip_balance,space_cache=v2,noatime,subvolid=0 0 0
>
>
> if I uncomment it, systemd hangs on trying to mount the FS with missing
> device
>
> sauron:~# cat /etc/crypttab
> cryptroot /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_2TB_S4Z7NG0M709677T-part7 /.cppwd luks,discard
> pool1 /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_Plus_2TB_S4Z7NG0M709677T-part8 /.cppwd luks,discard
>
> So two main I don't understand are:
> How can I get cryptdisks.service not to refuse to start at boot and tell me what dep it's missing?
> if my filesystems are in /etc/fstab, they fail to mount with
> Oct 27 22:22:59 sauron.svh.merlins.org systemd[1]: dev-disk-by\x2dlabel-btrfs_pool1.device: Job dev-disk-by\x2dlabel-btrfs_pool1.device/start timed out.
> Oct 27 22:22:59 sauron.svh.merlins.org systemd[1]: Timed out waiting for device dev-disk-by\x2dlabel-btrfs_pool1.device - /dev/disk/by-label/btrfs_pool1.
> Oct 27 22:22:59 sauron.svh.merlins.org systemd[1]: Dependency failed for mnt-btrfs_pool1.mount - /mnt/btrfs_pool1.
> Oct 27 22:22:59 sauron.svh.merlins.org systemd[1]: mnt-btrfs_pool1.mount: Job mnt-btrfs_pool1.mount/start failed with result 'dependency'.
> that amkes sense, but if that happens, I get the emergency prompt
> and at that point
> sauron:~# systemctl start cryptdisks.service
> does not work. It fails and I can't find any errors in journalctl -xb or anything
> about a missing dependency, although I assume that's why it won't start
>
> So I'm forced to comment out all of fstab, resume the boot, let it finish
> only then does
> sauron:~# systemctl start cryptdisks.service
> work
> After that I can uncomment fstab and mount the partitions
> So why was it refusing to start earlier, and it's ok now?
>
> I suppose this is not great:
> sauron:~# systemctl enable cryptdisks.service
> > Synchronizing state of cryptdisks.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
> > Executing: /usr/lib/systemd/systemd-sysv-install enable cryptdisks
> > The unit files have no installation config (WantedBy=, RequiredBy=, UpheldBy=,
> > Also=, or Alias= settings in the [Install] section, and DefaultInstance= for
> > template units). This means they are not meant to be enabled or disabled using systemctl.
> >
> > Possible reasons for having these kinds of units are:
> > • A unit may be statically enabled by being symlinked from another unit's
> > .wants/, .requires/, or .upholds/ directory.
> > • A unit's purpose may be to act as a helper for some other unit which has
> > a requirement dependency on it.
> > • A unit may be started when needed via activation (socket, path, timer,
> > D-Bus, udev, scripted systemctl call, ...).
> > • In case of template units, the unit is meant to be enabled with some
> > instance name specified.
>
> The autogenerated file below probably has bad deps, so removed the old initscript
> copied /lib/systemd/system/cryptdisks.service to /lib/systemd/system/cryptdisks.service
> then I had a look at systemctl list-dependencies
> and now have
> > isauron:~# cat /lib/systemd/system/cryptdisks.service
> > # Automatically generated by systemd-sysv-generator
> >
> > [Unit]
> > Documentation=man:systemd-sysv-generator(8)
> > SourcePath=/etc/init.d/cryptdisks
> > Description=LSB: Setup remaining encrypted block devices.
> > Before=local-fs.target
> >
> > [Service]
> > Type=forking
> > Restart=no
> > TimeoutSec=5min
> > IgnoreSIGPIPE=no
> > KillMode=process
> > GuessMainPID=no
> > RemainAfterExit=yes
> > SuccessExitStatus=5 6
> > ExecStart=/etc/init.d/cryptdisks start
> > ExecStop=/etc/init.d/cryptdisks stop
> > ExecReload=/etc/init.d/cryptdisks reload
>
> while the autogenerated one, was:
> > /lib/systemd/system/cryptdisks.service:
> > # Automatically generated by systemd-sysv-generator
> >
> > [Unit]
> > Documentation=man:systemd-sysv-generator(8)
> > SourcePath=/etc/init.d/cryptdisks
> > Description=LSB: Setup remaining encrypted block devices.
> > Before=checkfs.service
> > After=checkroot.service
> > After=cryptdisks-early.service
> > After=udev.service
> > After=mdadm-raid.service
> > After=lvm2.service
> >
> > [Service]
> > Type=forking
> > Restart=no
> > TimeoutSec=5min
> > IgnoreSIGPIPE=no
> > KillMode=process
> > GuessMainPID=no
> > RemainAfterExit=yes
> > SuccessExitStatus=5 6
> > ExecStart=/etc/init.d/cryptdisks start
> > ExecStop=/etc/init.d/cryptdisks stop
> > ExecReload=/etc/init.d/cryptdisks reload
> >
> > sauron:~# cat /etc/default/cryptdisks
> > # Run cryptdisks initscripts at startup? Default is Yes.
> > CRYPTDISKS_ENABLE=Yes
> >
> > # Mountpoints to mount, before cryptsetup is invoked at initscripts. Takes
> > # mountpoins which are configured in /etc/fstab as arguments. Separate
> > # mountpoints by space.
> > # This is useful for keyfiles on removable media. Default is unset.
> > CRYPTDISKS_MOUNT=""
> >
> > # Default check script. Takes effect, if the 'check' option is set in crypttab
> > # without a value.
> > CRYPTDISKS_CHECK=blkid
>
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
>
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
More information about the systemd-devel
mailing list