[systemd-devel] Raid device issue

[Ken Bass]

I just upgraded from F14 to F16 so this is my first attempt at systemd 
and I think I found a bug/issue. (I'm running systemd-37-3.fc16.x86_64)

My /dev/md0 (not a bootable disk thank goodness) is encrypted with luks.

My crypttab contains:
encrypt_storage  /dev/md0    /mnt/usbkey/storage.key 
cipher=aes-cbc-essiv:sha256,size=256,hash=sha1

systemd would fail with:
[   26.576425] systemd-cryptsetup[951]: crypt_init() failed: Block 
device required

That tells me that systemd is trying to access /dev/md0 before it 
actually created the device. So,
after much documentation reading I created a local  cryptsetup.target file:

[Unit]
Description=Encrypted Volumes Wait
Requires=mnt-usbkey.mount dev-md0.device
After=mnt-usbkey.mount dev-md0.device

The mnt-usbkey.mount ensures that the USB flash key is loaded early. 
(Before cryptsetup.target). That is not germane to this discussion.

The problem is the most of the time my changes fail. Occasionally it 
would work, but mostly not. So after examining the logs and some debug, 
I think the problem is a race with systemd or something.

The problem is dev-md0.device becomes 'plugged' before the /dev/md0 is 
actually created/ready. From the log:

[   26.136607] systemd[1]: dev-md0.device changed dead -> plugged
[   26.136614] systemd[1]: Job dev-md0.device/start finished, result=done
[   26.136634] systemd[1]: sys-devices-virtual-block-md0.device changed 
dead -> plugged
[   26.136691] systemd[1]: About to execute: 
/lib/systemd/systemd-cryptsetup attach encrypt_storage /dev/md0 
/mnt/usbkey/storage.key cipher=aes-cbc-essiv:sha256,size=256,hash=sha1
...
[   26.576425] systemd-cryptsetup[951]: crypt_init() failed: Block 
device required

However, the md0 is not really up. Because the above command only fails 
with /dev/md0 is missing. AND 500 milliseconds later...

[   26.634837] md/raid:md0: raid level 5 active with 4 out of 4 devices, 
algorithm 2
[   26.634839] RAID conf printout:
[   26.634840]  --- level:5 rd:4 wd:4
[   26.634842]  disk 0, o:1, dev:sdb1
[   26.634844]  disk 1, o:1, dev:sdc1
[   26.634845]  disk 2, o:1, dev:sdd1
[   26.634847]  disk 3, o:1, dev:sde1
[   26.634877] md0: detected capacity change from 0 to 2250456956928
...
[   26.672387] systemd[1]: 
dev-disk-by\x2did-md\x2duuid\x2d432b22e3:364555c6:f3927192:227cec7f.device 
changed dead -> plugged

So I came to the conclusion that I had to modifty crypttab to be:
encrypt_storage  
/dev/disk/by-id/md-uuid-432b22e3:364555c6:f3927192:227cec7f    
/mnt/usbkey/storage.key cipher=aes-cbc-essiv:sha256,size=256,hash=sha1

With that device change, it appears to work.

If this means dev-md0.device is not accurate as an indicator of /dev/md0 
being present, then I guess every other reference such as in my 
cryptsetup.target would need to be changed too.

This whole issue seems odd and certainly not intuitive. I thought I was 
instructing systemd correcty, telling it to have the depending on 
dev-md0.device. I would have thought any 'uuid' /dev/disk would simply 
be an alias, but that does not appear to be the case.

Note that my kernel cmdline has 'rd.md=0 rd.dm=0 rd.luks=0' so none of 
this is setup earlier than systemd.