[systemd-devel] removing dm-crypt mapping hangs on shutdown

[Dave Reisner]

On Wed, Jun 13, 2012 at 02:58:10PM +0100, Colin Guthrie wrote:
> 'Twas brillig, and Dave Reisner at 13/06/12 01:50 did gyre and gimble:
> > Hi,
> > 
> > I'm having some problems with a few of my VMs which all share in common
> > an encrypted root. They pivot into a shutdown ramfs and walk through the
> > blockdevs in sysfs, breaking down and unmounting each device.
> > Universally, these VMs all hang when calling 'cryptsetup remove', and
> > only under systemd. The relevant bits of stack trace look something like
> > this:
> > 
> > ioctl(3, DM_DEV_REMOVE, 0xe1b8b0)       = 0
> > semget(0xd4d255f, 1, 0)                 = 229376
> > semctl(229376, 0, GETVAL, 0xffffffffffffffff) = 2
> > semop(229376, {{0, -1, IPC_NOWAIT}}, 1) = 0
> > semop(229376, {{0, 0, 0}}, 1
> 
> I've seen similar symptoms in bug reports to your experience. Basically
> hanging after detaching dm devices.

For whatever reason, I've only started seeing this recently, but I'm not
surprised that others are seeing it as well.

> > I've found 2 solutions so far to avoiding this hang:
> > 
> > 1) Not letting systemd get its paws on my block devices to umount them.
> > Specifically, avoiding the dm_detach_all() call seems to alleviate this.
> 
> Yeah, it wouldn't be right to avoid it completely, but I agree that it
> does need to skip things a bit more robustly.
> 
> > So this, to me, raises 2 questions:
> > 
> > 1) Can systemd just leave block devices which it didn't assemble alone?
> > I'd expect that if the initramfs assembled and mounted a device, it
> > should be responsible for tearing it down. In theory, this could just be
> > something as simple as just skipping over / and /usr.
> 
> I don't think a simply solution would work here as there are other cases
> that may trigger the initramfs to enable certain devices (e.g. resume
> from an encrypted swap partition for example).

Sure, there's already code to parse /proc/self/mountinfo and
/proc/swaps. I don't know offhand how early this runs, but there's an
opportunity to do some accounting here and mark already active devices
as "off limits" for disassembly/unmount on shutdown.

Alternatively, rather than using a brute force approach, use something a
little smarter which involves the holders attribute of any given block
device in /sys/class/block. You can easily recurse down the chain of
child devices and disassemble as the stack unwinds. In shell script, it
looks something like this:

  http://projects.archlinux.org/mkinitcpio.git/tree/shutdown#n44

> I guess the initramfs should really write something in /run/initramfs
> folder that indicates which devices it's "managing" such that systemd
> can only deal with the remainder.
> 
> 
> I'm not sure what the current status is, but I wonder what would happen
> when systemd tries to unmount /usr just now... :s
> 

Nothing interesting, just an EBUSY.

Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounted /dev/mqueue.
Unmounted /dev/hugepages.
Unmounted /sys/kernel/debug.
Could not unmount /usr: Device or resource busy
Disabling swaps.
Detaching loop devices.
Detaching DM devices.
Successfully changed into root pivot.
Detaching loop devices.
Unmounting all devices.
Disassembling stacked devices.
[   28.580318] Power down.

> 
> > 2) What's the goal of calling mlockall() here? The original patch that
> > added this (b1b2a107d15a) gives no indication of why it's wanted/needed.
> 
> No idea on this one.
> 
> Col
> 
> 
> -- 
> 
> Colin Guthrie
> gmane(at)colin.guthr.ie
> http://colin.guthr.ie/
> 
> Day Job:
>   Tribalogic Limited http://www.tribalogic.net/
> Open Source:
>   Mageia Contributor http://www.mageia.org/
>   PulseAudio Hacker http://www.pulseaudio.org/
>   Trac Hacker http://trac.edgewall.org/
> 
> 
> 
> _______________________________________________
> systemd-devel mailing list
> systemd-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/systemd-devel