[systemd-devel] Mounting a new device to a mount point with an old (auto-generated) but inactive mount unit triggers an immediate unmount

[Silvio Knizek]

Am Mittwoch, dem 07.07.2021 um 15:21 +0200 schrieb Christian Rohmann:
> Hello systemd-devel,
>
>
> we recently replaced a broken drive on a server and ran into a strange
> issue in regards to a mount ....
>
>
>
> 1) It started with the old device not being reachable anymore and
> therefore the crypt setup and mount was just failing:
>
> --- cut ---
> [...]
>
> Jul  2 03:41:11 myserver systemd[1]:
> dev-disk-by\x2duuid-ee386599\x2d8235\x2d4d4d\x2d9d3e\x2d901ccf2eed4b.device:
> Job
> dev-disk-by\x2duuid-ee386599\x2d8235\x2d4d4d\x2d9d3e\x2d901ccf2eed4b.device/start
> timed out.
> Jul  2 03:41:11 myserver systemd[1]: Timed out waiting for device
> dev-disk-by\x2duuid-ee386599\x2d8235\x2d4d4d\x2d9d3e\x2d901ccf2eed4b.device.
> Jul  2 03:41:11 myserver systemd[1]: Dependency failed for Cryptography
> Setup for ee386599-8235-4d4d-9d3e-901ccf2eed4b_crypt.
> Jul  2 03:41:11 myserver systemd[1]:
> systemd-cryptsetup at ee386599\x2d8235\x2d4d4d\x2d9d3e\x2d901ccf2eed4b_crypt.service:
> Job
> systemd-cryptsetup at ee386599\x2d8235\x2d4d4d\x2d9d3e\x2d901ccf2eed4b_crypt.service/start
> failed with result 'dependency'.
> Jul  2 03:41:11 myserver systemd[1]:
> dev-disk-by\x2duuid-ee386599\x2d8235\x2d4d4d\x2d9d3e\x2d901ccf2eed4b.device:
> Job
> dev-disk-by\x2duuid-ee386599\x2d8235\x2d4d4d\x2d9d3e\x2d901ccf2eed4b.device/start
> failed with result 'timeout'.
> Jul  2 03:41:11 myserver systemd[1]:
> dev-ceph\x2dee386599\x2d8235\x2d4d4d\x2d9d3e\x2d901ccf2eed4b-data.device:
> Job
> dev-ceph\x2dee386599\x2d8235\x2d4d4d\x2d9d3e\x2d901ccf2eed4b-data.device/start
> timed out.
> Jul  2 03:41:11 myserver systemd[1]: Timed out waiting for device
> dev-ceph\x2dee386599\x2d8235\x2d4d4d\x2d9d3e\x2d901ccf2eed4b-data.device.
> Jul  2 03:41:11 myserver systemd[1]: Dependency failed for
> /var/lib/ceph/osd/ceph-64.
> Jul  2 03:41:11 myserver systemd[1]: var-lib-ceph-osd-ceph\x2d64.mount:
> Job var-lib-ceph-osd-ceph\x2d64.mount/start failed with result 'dependency'.
> Jul  2 03:41:11 myserver systemd[1]:
> dev-ceph\x2dee386599\x2d8235\x2d4d4d\x2d9d3e\x2d901ccf2eed4b-data.device:
> Job
> dev-ceph\x2dee386599\x2d8235\x2d4d4d\x2d9d3e\x2d901ccf2eed4b-data.device/start
> failed with result 'timeout'.
>
> [...]
> --- cut ---
>
>
> 2) We then replaced the drive, created a new Luks device and created a
> new XFS filesystem on top of it.
>
>
> 3) We then updated the fstab to point to the new device name (ceph-$UUID
> of the new luks device) and triggered am immediate "mount" via Ansible
> The fstab entry now looks like this:
>
> --- cut ---
> /dev/ceph-8ce1a4e6-94dd-4fa8-a7b9-310ab211b54a/data
> /var/lib/ceph/osd/ceph-64 xfs nofail 0 0
>
> --- cut ---
>
>
> and the systemd auto-generated mount unit like so:
>
> --- cut ---
>
> # /run/systemd/generator/var-lib-ceph-osd-ceph\x2d64.mount
> # Automatically generated by systemd-fstab-generator
>
> [Unit]
> SourcePath=/etc/fstab
> Documentation=man:fstab(5) man:systemd-fstab-generator(8)
>
> [Mount]
> Where=/var/lib/ceph/osd/ceph-64
> What=/dev/ceph-8ce1a4e6-94dd-4fa8-a7b9-310ab211b54a/data
> Type=xfs
> Options=nofail
> --- cut ---
>
>
> What happend was, that the mount succeeded initially, but then systemd
> unmounted the path right away,
> apparently because the old (auto-generated) mount unit was still
> present, in "inactive" state:
>
> --- cut ---
>
> Jul  2 13:17:04 myserver ansible-mount: Invoked with
> src=/dev/ceph-8ce1a4e6-94dd-4fa8-a7b9-310ab211b54a/data
> path=/var/lib/ceph/osd/ceph-64 fstype=xfs state=mounted opts=nofail
> boot=True backup=False dump=None fstab=None passno=None
> Jul  2 13:17:04 myserver kernel: [367734.361658] XFS (dm-25): Mounting
> V5 Filesystem
> Jul  2 13:17:04 myserver kernel: [367734.373636] XFS (dm-25): Ending
> clean mount
> Jul  2 13:17:04 myserver systemd[1]: var-lib-ceph-osd-ceph\x2d64.mount:
> Unit is bound to inactive unit
> dev-ceph\x2dee386599\x2d8235\x2d4d4d\x2d9d3e\x2d901ccf2eed4b-data.device.
> Stopping, too.
> Jul  2 13:17:04 myserver systemd[1]: Unmounting /var/lib/ceph/osd/ceph-64...
> Jul  2 13:17:05 myserver systemd[1]: Unmounted /var/lib/ceph/osd/ceph-64.
> Jul  2 13:17:05 myserver kernel: [367734.413530] XFS (dm-25): Unmounting
> Filesystem
>
> --- cut ---
>
>
>
> I did find the issue https://github.com/systemd/systemd/issues/1741
> which sounds quite similar, but that is closed for comments unfortunately.
> There a supposed "workaround" apparently is to call "systemd
> daemon-reload" prior to mounting? Is this the proper way to update the
> auto-generated mount-units then?
>
> Or did we handle this whole process of replacing a device and then
> having a new one mounted at the same mount point wrong in any way?
> As explained, we were not handling any systemd mount units directly, but
> only modified the fstab.
>
>
>
> Thanks and with kind Regards
>
>
> Christian
Hi Christian,

after touching /etc/fstab you're supposed to run `systemctl daemon-
reload` to re-trigger the generators. This is in fact a feature to
announce changes in configuration files to systemd. See
man:systemd.generator for more information.

BR
Silvio