[systemd-devel] systemd-cryptsetup device requires a valid and unique target UUID - breaks btrfs multivolume?

[Anders Andersson]

Hi! I have been trying to set up the "perfect" (for me) encrypted
server for some time now, and I have been running into a lot of
issues. After extensive digging and trying to learn about systemd, I
think I have found the root cause, and I would like some input from
people with more experience. Please excuse the verbosity of this
email, but it is essentially a detailed bug report containing what I
hope is every little detail that is required to recreate and debug
this issue.

First some background: The problems arise when I want to run a
multi-device btrfs filesystem on LUKS encrypted backing devices, and
use a keyfile that resides on another encrypted filesystem. Since I am
a Debian user I would have previously used the
keyscript=decrypt_derived method to solve this, but now I want to use
systemd, where this is missing. The problem is that I get a lot of
timeouts for no apparent reason, and I have not managed to tell
systemd about the dependencies, and they keep timing out even though
they are obviously correctly unlocked.

After debugging this for a while I think it comes down to that when
systemd tries to unlock an encrypted block device and create a
corresponding /dev/mapper/foo and a dev-mapper-foo.device, it fails if
the unlocked device does not have a valid UUID. It will successfully
unlock the device and create the correct mapping, but then it hangs,
and reports the device as inactive (dead).

I will explain what I expect to work, and what does not work, with
these commands. This is on a Fedora Server virtual machine, which I
have never tried before but installed just to test this!

I have created a partition on a block device (/dev/sdb1) which I have
prepared with luksFormat using a keyfile called "/keyfile", located on
the root filesystem. Nothing fancy here. I put a simple entry in
/etc/crypttab and had systemctl daemon-reload generate the
corresponding files in /var/run/systemd/generator, which I copied to
/etc/systemd/system/.


-------- /etc/systemd/system/systemd-cryptsetup at crypt.service --------
# Automatically generated by systemd-cryptsetup-generator

[Unit]
Description=Cryptography Setup for %I
Documentation=man:crypttab(5) man:systemd-cryptsetup-generator(8)
man:systemd-cryptsetup at .service(8)
SourcePath=/etc/crypttab
DefaultDependencies=no
Conflicts=umount.target
BindsTo=dev-mapper-%i.device
IgnoreOnIsolate=true
After=cryptsetup-pre.target
Before=cryptsetup.target
RequiresMountsFor=/keyfile
BindsTo=dev-sdb1.device
After=dev-sdb1.device
Before=umount.target

[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutSec=0
ExecStart=/usr/lib/systemd/systemd-cryptsetup attach 'crypt'
'/dev/sdb1' '/keyfile' 'luks'
ExecStop=/usr/lib/systemd/systemd-cryptsetup detach 'crypt'
-------- /etc/systemd/system/systemd-cryptsetup at crypt.service --------


There is also a link from dev-mapper-crypt.device.requires:

dev-mapper-crypt.device.requires
 └── systemd-cryptsetup at crypt.service -> ../systemd-cryptsetup at crypt.service






With this set up, and the encrypted target block device has a plain
old ext4 filesystem, everything works like I expect it to:

[root at fedora ~]# systemctl status dev-mapper-crypt.device
● dev-mapper-crypt.device
   Loaded: loaded
   Active: inactive (dead)
[root at fedora ~]# systemctl start dev-mapper-crypt.device
[root at fedora ~]# systemctl status dev-mapper-crypt.device
● dev-mapper-crypt.device - /dev/mapper/crypt
   Follow: unit currently follows state of
sys-devices-virtual-block-dm\x2d2.device
   Loaded: loaded
   Active: active (plugged) since Sat 2015-12-12 14:12:44 CET; 6s ago
   Device: /sys/devices/virtual/block/dm-2



This means that systemd knows about dev-mapper-crypt.device, it can
start automatically if wanted, and it can be used as a dependency.
Great!

Now I make a change. I remove the ext4 filesystem from the encrypted
target by filling /dev/mapper/crypt with zeros, reboot for good
measure, and try this again:

root at fedora ~]# systemctl status dev-mapper-crypt.device
● dev-mapper-crypt.device
   Loaded: loaded
   Active: inactive (dead)
[root at fedora ~]# systemctl start dev-mapper-crypt.device

... and here it stops. Nothing happens. If I log in again and check
what's going on I see this:
 1197 pts/0    Ss     0:00  |       \_ -bash
 1219 pts/0    S+     0:00  |           \_ systemctl start
dev-mapper-crypt.device
 1220 pts/0    S+     0:00  |               \_
/usr/bin/systemd-tty-ask-password-agent --watch
 1221 pts/0    Sl+    0:00  |               \_ /usr/bin/pkttyagent
--notify-fd 5 --fallback

But there are no password prompts anywhere, not in any SSH sessions
nor in the actual console. Eventually the original command times out:

Job for dev-mapper-crypt.device timed out.
[root at fedora ~]# systemctl status dev-mapper-crypt.device
● dev-mapper-crypt.device
   Loaded: loaded
   Active: inactive (dead)

Dec 12 14:34:44 fedora.localdomain systemd[1]:
dev-mapper-crypt.device: Job dev-mapper-crypt.device/start timed out.
Dec 12 14:34:44 fedora.localdomain systemd[1]: Timed out waiting for
device dev-mapper-crypt.device.
Dec 12 14:34:44 fedora.localdomain systemd[1]:
dev-mapper-crypt.device: Job dev-mapper-crypt.device/start failed with
result 'timeout'.

However, /dev/mapper/crypt is live and well:

[root at fedora ~]# ls -l /dev/mapper/crypt
lrwxrwxrwx. 1 root root 7 Dec 12 14:33 /dev/mapper/crypt -> ../dm-2



Now, the reason I think that this is a problem with the UUID is that
when I have tried this with a multi-volume btrfs filesystem, I have
been getting the same type of timeouts on all devices but one, and
they have been happening in a "random" order. Btrfs assigns the same
UUID to all volumes participating in the filesystem, and this is what
leads me to believe that the problem is with that.

If anyone has managed to read this far, I thank you for the attention,
and I have two questions for you:

1) Am I right in assuming that a valid target UUID should not be in
any way necessary or relevant for unlocking an encrypted device? In my
eyes it is perfectly valid to have a device filled with for example
random data, filled with zeros, or with any other unknown pattern.

2) Is my understanding of systemd devices and services correct, in
that it should be possible to treat dev-mapper-crypt.device as a
service that I can start and stop using systemctl?



// Anders