[systemd-devel] Raid device issue

Thu Dec 8 12:45:52 PST 2011

On 12/08/2011 06:35 PM, Ken Bass wrote:
> I just want to add one additional potential race issue that I saw in my
> log files.
>
> I used the cryptsetup.target which I understood to be a synchronization
> barrier.
>
> In my case I used 'Before=cryptsetup.target' in my mnt-usbkey.mount, but
> this issue is more general.
>
> Looking at the log files, it appears that cryptsetup.target is triggered
> not triggered when the cryptsetup stuff begins.
> Rather, a separate service for each entry in /etc/crypttab completes
> right before cryptsetup.target.
>
> So I'm thinking that this is another case of a race issue. If someone
> specifies 'Before=cryptsetup.target' as a synchronization target, there
> is a window of time before that where stuff in the /etc/crypttab is
> being performed yet cryptsetup.target has not triggered.
Hi Ken,

maybe I'm misunderstanding you, but I think that you are looking at the 
meaning of cryptsetup.target in a wrong way: cryptsetup.target comes 
after all encrypted devices configured in /etc/crypttab are ready. So it 
makes sense to do something with After=cryptsetup.target (== after all 
encrypted devices have been opened), but not with Before=cryptsetup.target.

I guess that you might have to declare the ordering dependence between 
mnt-subkey.mount with Before=cryptsetup at encrypt_storage.service directly.

As far as your previous problems goes...
> The problem is dev-md0.device becomes 'plugged' before the /dev/md0 is actually created/ready. From the log:
>
> [   26.136607] systemd[1]: dev-md0.device changed dead -> plugged
> [   26.136614] systemd[1]: Job dev-md0.device/start finished, result=done
> [   26.136634] systemd[1]: sys-devices-virtual-block-md0.device changed dead -> plugged
> [   26.136691] systemd[1]: About to execute: /lib/systemd/systemd-cryptsetup attach encrypt_storage /dev/md0 /mnt/usbkey/storage.key cipher=aes-cbc-essiv:sha256,size=256,hash=sha1
> ...
> [   26.576425] systemd-cryptsetup[951]: crypt_init() failed: Block device required
>
> However, the md0 is not really up. Because the above command only fails with /dev/md0 is missing. AND 500 milliseconds later...
>
> [   26.634837] md/raid:md0: raid level 5 active with 4 out of 4 devices, algorithm 2
> [   26.634839] RAID conf printout:
> [   26.634840]  --- level:5 rd:4 wd:4
> [   26.634842]  disk 0, o:1, dev:sdb1
> [   26.634844]  disk 1, o:1, dev:sdc1
> [   26.634845]  disk 2, o:1, dev:sdd1
> [   26.634847]  disk 3, o:1, dev:sde1
> [   26.634877] md0: detected capacity change from 0 to 2250456956928
> ...
> [   26.672387] systemd[1]: dev-disk-by\x2did-md\x2duuid\x2d432b22e3:364555c6:f3927192:227cec7f.device changed dead -> plugged
... my mdadm knowledge is very rusty, but I think that /dev/md0 being 
present doesn't really mean that the array is ready. Completely 
independently of systemd, you can have a partially built array (which 
cannot be read) with the device file present. So again, it might not be 
possible to meaningfully wait for /dev/md0.

Best,
Zbyszek

> [ 27.073983] systemd[1]: About to execute:
> /lib/systemd/systemd-cryptsetup attach encrypt_storage
> /dev/disk/by-id/md-uuid-432b22e3:364555c6:f3927192:227cec7f
> /mnt/usbkey/storage.key cipher=aes-cbc-essiv:sha256,size=256,hash=sha1
> [ 27.086057] rmmod[1004]: ERROR: Module scsi_wait_scan does not exist in
> /proc/modules
> [ 27.104056] systemd[1]: Forked /lib/systemd/systemd-cryptsetup as 1007
> [ 27.104133] systemd[1]: cryptsetup at encrypt_storage.service changed dead
> -> start
> ...
> [ 29.042725] systemd[1]: cryptsetup at encrypt_storage.service: main
> process exited, code=exited, status=0
> [ 29.075045] systemd[1]: cryptsetup at encrypt_storage.service changed
> start -> exited
> [ 29.075051] systemd[1]: Job cryptsetup at encrypt_storage.service/start
> finished, result=done
> [ 29.075215] systemd[1]: cryptsetup.target changed dead -> active
> [ 29.075221] systemd[1]: Job cryptsetup.target/start finished, result=done
>
>  From the above logs, there is a full 2 seconds between systemd doing
> the '/lib/systemd/systemd-cryptsetup attach' and when 'cryptsetup.target
> changed dead -> active' is declared.
>
> In my case, I specified a 'Before=cryptsetup.target' in my
> mnt-usbkey.mount. However, what would prevent the mnt-usbkey.mount from
> not being executed within the above 2 second interval? This looks wrong
> to me.
> _______________________________________________
> systemd-devel mailing list
> systemd-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/systemd-devel
>