[systemd-devel] removing dm-crypt mapping hangs on shutdown

[Lennart Poettering]

On Tue, 12.06.12 20:50, Dave Reisner (d at falconindy.com) wrote:

> Hi,

Heya,

> I'm having some problems with a few of my VMs which all share in common
> an encrypted root. They pivot into a shutdown ramfs and walk through the
> blockdevs in sysfs, breaking down and unmounting each device.
> Universally, these VMs all hang when calling 'cryptsetup remove', and
> only under systemd. The relevant bits of stack trace look something like
> this:
> 
> ioctl(3, DM_DEV_REMOVE, 0xe1b8b0)       = 0
> semget(0xd4d255f, 1, 0)                 = 229376
> semctl(229376, 0, GETVAL, 0xffffffffffffffff) = 2
> semop(229376, {{0, -1, IPC_NOWAIT}}, 1) = 0
> semop(229376, {{0, 0, 0}}, 1

Well, the DM libraries for some weird reason find it funny to use SysV
IPC for intra-thread locking. I think it's quite a bad choice to use
SysV IPC these days. (And it also uses threads in the first place, where
it better shouldn't).

Anyway, most likely you have more than one thread running there and
your fg thread is waiting for the other one.

> I've found 2 solutions so far to avoiding this hang:
> 
> 1) Not letting systemd get its paws on my block devices to umount them.
> Specifically, avoiding the dm_detach_all() call seems to alleviate
> this.

Well, devices needed for the root fs should stay unnefected by
dm_detach_all() [as DM_DEV_REMOVE should return EBUSY in that case] and
the other ones should have been shutdown before we actually enter this
code. If that doesn't happen for you there is something wrong with the
destruction of LUKS media.

> 2) Removing the mlockall() call in src/core/shutdown.c on line 351
> (according to git master as of this posting).

Uh? not sure what that should change?

> So this, to me, raises 2 questions:
> 
> 1) Can systemd just leave block devices which it didn't assemble alone?
> I'd expect that if the initramfs assembled and mounted a device, it
> should be responsible for tearing it down. In theory, this could just be
> something as simple as just skipping over / and /usr.

We generally don't try to work around problems, we fix them where they
are. The shutdown tool is mostly just intended as last fallback if the
normal code didn't have the desired effect. It should leave the root fs
untouched...

> 2) What's the goal of calling mlockall() here? The original patch that
> added this (b1b2a107d15a) gives no indication of why it's wanted/needed.

It's merely an optimization to avoid that the shutdown process is
swapped out.

Lennart

-- 
Lennart Poettering - Red Hat, Inc.