[systemd-devel] fstab generator/capture.mount randomly fails which depend of cryptsetup.target

Thu Jan 16 09:00:23 UTC 2025

How to unsubscribe from here?

On Thu, Jan 16, 2025, 2:56 PM Sumit Kumar <sumit.kumar1 at trellix.com> wrote:

> Hi ,
>
> Our Linux based system uses an external storage disk called capture.
> The capture disk is encrypted with LUKS to protect stored data. During
> system boot, system uses the LUKS utilities to decrypt the disk before
> mounting the file systems containing the stored data. Due to an issue with
> inconsistent ordering of decryption and mounting services during system
> bootup, during reboots the system with capture disk can get into emergency
> mode.
>
> Do we have any configuration problem ?
>
> *System details, logs and unit files details : *
> systemd version: 252
> Used distribution : AlmaLinux 9.4
> Linux kernel version used : 5.15.160-1.11.10.0.x86_64
> Journalctl -xb output for error message:
>
> Jan 10 09:53:04 ps systemd[1]: Listening on udev Control Socket.
> Jan 10 09:53:04 ps systemd[1]: Listening on udev Kernel Socket.
> Jan 10 09:53:04 ps systemd[1]: capture.mount: Directory /capture to mount over is not empty, mounting anyway.
> Jan 10 09:53:04 ps systemd[1]: Mounting /capture...
> Jan 10 09:53:04 ps systemd[1]: Mounting Huge Pages File System...
> Jan 10 09:53:04 ps systemd[1]: Mounting POSIX Message Queue File System...
> Jan 10 09:53:04 ps systemd[1]: Mounting /sys/kernel/debug...
> Jan 10 09:53:04 ps systemd[1]: Mounting Kernel Trace File System...
> Jan 10 09:53:04 ps systemd[1]: Kernel Module supporting RPCSEC_GSS was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab).
> Jan 10 09:53:04 ps systemd[1]: Starting Availability of block devices...
> Jan 10 09:53:04 ps systemd[1]: Starting Create List of Static Device Nodes...
> Jan 10 09:53:04 ps systemd[1]: Starting Load legacy module configuration...
> Jan 10 09:53:04 ps systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
> Jan 10 09:53:04 ps systemd[1]: Starting Load Kernel Module configfs...
> Jan 10 09:53:04 ps systemd[1]: Starting Load Kernel Module drm...
> Jan 10 09:53:04 ps systemd[1]: Starting Load Kernel Module efi_pstore...
> Jan 10 09:53:04 ps systemd[1]: Starting Load Kernel Module fuse...
> Jan 10 09:53:04 ps systemd[1]: Starting Read and set NIS domainname from /etc/sysconfig/network...
> Jan 10 09:53:04 ps systemd[1]: First Boot Wizard was skipped because of an unmet condition check (ConditionFirstBoot=yes).
> Jan 10 09:53:04 ps systemd[1]: Rebuild Hardware Database was skipped because of an unmet condition check (ConditionNeedsUpdate=/etc).
> Jan 10 09:53:04 ps systemd[1]: systemd-journald.service: unit configures an IP firewall, but the local system does not support BPF/cgroup firewalling.
> Jan 10 09:53:04 ps systemd[1]: (This warning is only shown for the first unit using IP firewalling.)
> Jan 10 09:53:04 ps systemd[1]: Starting Journal Service...
> Jan 10 09:53:04 ps systemd[1]: Starting Load Kernel Modules...
> Jan 10 09:53:04 ps systemd[1]: Starting Generate network units from Kernel command line...
> Jan 10 09:53:04 ps systemd[1]: TPM2 PCR Machine ID Measurement was skipped because of an unmet condition check (ConditionPathExists=/sys/firmware/efi/efivars/StubPcrKernelImage-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f).
> Jan 10 09:53:04 ps systemd[1]: Create System Users was skipped because no trigger condition checks were met.
> Jan 10 09:53:04 ps systemd[1]: Starting Coldplug All udev Devices...*Jan 10 09:53:04 ps systemd[1]: capture.mount: Mount process exited, code=exited, status=32/n/a
> Jan 10 09:53:04 ps systemd[1]: capture.mount: Failed with result 'exit-code'*.*Jan 10 09:53:04 ps systemd[1]: Failed to mount /capture.*
> Jan 10 09:53:04 ps systemd[1]: Dependency failed for Local File Systems.
> Jan 10 09:53:04 ps systemd[1]: Dependency failed for Systemd service ordering.
> Jan 10 09:53:04 ps systemd[1]: service-ordering.service: Job service-ordering.service/start failed with result 'dependency'.
> Jan 10 09:53:04 ps systemd[1]: Dependency failed for Create Volatile Files and Directories.
> Jan 10 09:53:04 ps systemd[1]: systemd-tmpfiles-setup.service: Job systemd-tmpfiles-setup.service/start failed with result 'dependency'.
> Jan 10 09:53:04 ps systemd[1]: Dependency failed for Mark the need to relabel after reboot.
> Jan 10 09:53:04 ps systemd[1]: selinux-autorelabel-mark.service: Job selinux-autorelabel-mark.service/start failed with result 'dependency'.
> Jan 10 09:53:04 ps systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'.
> Jan 10 09:53:04 ps systemd[1]: local-fs.target: Triggering OnFailure= dependencies.
> Jan 10 09:53:04 ps systemd[1]: Unnecessary job was removed for /dev/ttyS0.
> Jan 10 09:53:04 ps systemd[1]: Mounted Huge Pages File System.
>
> *systemctl list-units --failed*
>
>   UNIT          LOAD   ACTIVE SUB    DESCRIPTION* capture.mount loaded failed failed /capture
>
> LOAD   = Reflects whether the unit definition was properly loaded.
> ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
> SUB    = The low-level unit activation state, values depend on unit type.
> 1 loaded units listed.
> *systemctl status capture.mount*
> x capture.mount - /capture
>      Loaded: loaded (/etc/fstab; generated)
>      Active: failed (Result: exit-code) since Fri 2025-01-10 09:53:03 UTC; 39min ago
>       Where: /capture
>        What: /dev/mapper/encr-sdd
>        Docs: man:fstab(5)
>              man:systemd-fstab-generator(8)
>         CPU: 5ms
>
> Jan 10 09:53:04 ps mount[784]: mount: /capture: special device /dev/mapper/encr-sdd does not exist.
> Notice: journal has been rotated since unit was started, output may be incomplete.
>
> *systemctl cat capture.mount*# /run/systemd/generator/capture.mount# Automatically generated by systemd-fstab-generator
>
> [Unit]
> Documentation=man:fstab(5) man:systemd-fstab-generator(8)
> SourcePath=/etc/fstab
> After=cryptsetup.target
> Requires=cryptsetup.target
> Before=local-fs.target
> Requires=systemd-fsck at dev-mapper-encr\x2dsdd.service
> After=systemd-fsck at dev-mapper-encr\x2dsdd.service
> After=blockdev at dev-mapper-encr\x2dsdd.target
>
> [Mount]
> What=/dev/mapper/encr-sdd
> Where=/capture
> Type=ext4
> Options=defaults,nosuid,x-systemd.requires=cryptsetup.target
> *cat /etc/crypttab*
> encr-sdd UUID=6af9171e-82c7-4be2-bd70-14979067727f /.encr/sdd
>
> cat /etc/fstab
> /dev/disk/by-label/boot            /boot			vfat ro,defaults	1 2
> /dev/disk/by-label/opt           /opt            ext4 ro,defaults    1 2
> /dev/disk/by-label/root           /            ext4 ro,defaults    1 1
> /dev/disk/by-label/notused           /notused            ext4 defaults,ro,noauto    0 0
> /dev/disk/by-label/rescue           /rescue            ext4 ro,defaults,noauto    0 0
> /dev/disk/by-label/var           /var            ext4 defaults    1 2
> /dev/disk/by-label/var_log           /var/log            ext4 defaults    1 2
> /dev/disk/by-label/config           /config            ext4 defaults,nosuid    1 2
> /dev/disk/by-label/swap               swap                    swap    defaults        */dev/mapper/encr-sdd           /capture            ext4 defaults,nosuid,x-systemd.requires=cryptsetup.target    1 2*
> /dev/mapper/v0-deferred			/deferred			ext4 defaults,noatime,noexec,nosuid		1 3
> /dev/mapper/v0-scandir			/scandir			ext4 defaults,noatime,noexec,nosuid,nobarrier		1 3
> /dev/mapper/v0-logs			/logs			ext4 defaults,noatime,noexec,nosuid		1 3
> /dev/mapper/v0-wk			/wk			ext4 defaults,nosuid		1 3
> /var/etc   /etc			none bind,x-systemd.requires-mounts-for=/var 0	0 0
> /wk/tmp   /tmp			none bind,x-systemd.requires-mounts-for=/wk 0	0 0
> /wk/root   /root			none bind,x-systemd.requires-mounts-for=/wk 0	0 0
> /wk/home   /home			none bind,x-systemd.requires-mounts-for=/wk 0	0 0
> /wk/AgentDB/db   /var/McAfee/agent/db			none bind,x-systemd.requires-mounts-for=/wk 1	0 0
> /dev/cdrom /media/cdrom iso9660 noexec,nosuid,noauto,owner,ro 0 0
> /dev/cdrom-0 /media/cdrom-0 iso9660 noexec,nosuid,noauto,owner,ro 0 0
> /dev/cdrom-usb /media/cdrom-usb iso9660 noexec,nosuid,noauto,owner,ro 0 0
> none			/proc/fs/nfsd		nfsd	noauto		0 0
> nodev /sys/kernel/debug    debugfs   defaults   0  0
>
> *systemctl list-unit-files | grep systemd-cryptsetup*
> systemd-cryptsetup at encr\x2dsdd.service     generated       -
> *systemctl status systemd-cryptsetup at encr\x2dsdd.service ** systemd-cryptsetup at encr\x2dsdd.service - Cryptography Setup for encr-sdd
>      Loaded: loaded (/etc/crypttab; generated)
>      Active: inactive (dead)
>        Docs: man:crypttab(5)
>              man:systemd-cryptsetup-generator(8)
>              man:systemd-cryptsetup at .service(8)
>
> *cryptsetup status encr-sdd*
> /dev/mapper/encr-sdd is inactive.
> *ls -lrt /dev/mapper/**
> crw------- 1 root root 10, 236 Jan 10 09:53 /dev/mapper/control
> lrwxrwxrwx 1 root root       7 Jan 10 09:53 /dev/mapper/v0-deferred -> ../dm-0
> lrwxrwxrwx 1 root root       7 Jan 10 09:53 /dev/mapper/v0-scandir -> ../dm-1
> lrwxrwxrwx 1 root root       7 Jan 10 09:53 /dev/mapper/v0-wk -> ../dm-3
> lrwxrwxrwx 1 root root       7 Jan 10 09:53 /dev/mapper/v0-logs -> ../dm-2
>
>
> systemctl show systemd-cryptsetup at encr\x2dsdd.service
> Type=oneshot
> ExitType=main
> Restart=no
> NotifyAccess=none
> RestartUSec=100ms
> TimeoutStartUSec=infinity
> TimeoutStopUSec=infinity
> TimeoutAbortUSec=infinity
> TimeoutStartFailureMode=terminate
> TimeoutStopFailureMode=terminate
> RuntimeMaxUSec=infinity
> RuntimeRandomizedExtraUSec=0
> WatchdogUSec=infinity
> WatchdogTimestampMonotonic=0
> RootDirectoryStartOnly=no
> RemainAfterExit=yes
> GuessMainPID=yes
> MainPID=0
> ControlPID=0
> FileDescriptorStoreMax=0
> NFileDescriptorStore=0
> StatusErrno=0
> Result=success
> ReloadResult=success
> CleanResult=success
> UID=[not set]
> GID=[not set]
> NRestarts=0
> OOMPolicy=stop
> ReloadSignal=1
> ExecMainStartTimestampMonotonic=0
> ExecMainExitTimestampMonotonic=0
> ExecMainPID=0
> ExecMainCode=0
> ExecMainStatus=0
> ExecStart={ path=/usr/lib/systemd/systemd-cryptsetup ; argv[]=/usr/lib/systemd/systemd-cryptsetup attach encr-sdd /dev/disk/by-uuid/6af9171e-82c7-4be2-bd70-14979067727f /.encr/sdd  ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
> ExecStartEx={ path=/usr/lib/systemd/systemd-cryptsetup ; argv[]=/usr/lib/systemd/systemd-cryptsetup attach encr-sdd /dev/disk/by-uuid/6af9171e-82c7-4be2-bd70-14979067727f /.encr/sdd  ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
> ExecStop={ path=/usr/lib/systemd/systemd-cryptsetup ; argv[]=/usr/lib/systemd/systemd-cryptsetup detach encr-sdd ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
> ExecStopEx={ path=/usr/lib/systemd/systemd-cryptsetup ; argv[]=/usr/lib/systemd/systemd-cryptsetup detach encr-sdd ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
> Slice=system-systemd\x2dcryptsetup.slice
> ControlGroupId=0
> MemoryCurrent=[not set]
> MemoryAvailable=infinity
> CPUUsageNSec=[not set]
> TasksCurrent=[not set]
> IPIngressBytes=[no data]
> IPIngressPackets=[no data]
> IPEgressBytes=[no data]
> IPEgressPackets=[no data]
> IOReadBytes=18446744073709551615
> IOReadOperations=18446744073709551615
> IOWriteBytes=18446744073709551615
> IOWriteOperations=18446744073709551615
> Delegate=no
> CPUAccounting=yes
> CPUWeight=[not set]
> StartupCPUWeight=[not set]
> CPUShares=[not set]
> StartupCPUShares=[not set]
> CPUQuotaPerSecUSec=infinity
> CPUQuotaPeriodUSec=infinity
> IOAccounting=no
> IOWeight=[not set]
> StartupIOWeight=[not set]
> BlockIOAccounting=no
> BlockIOWeight=[not set]
> StartupBlockIOWeight=[not set]
> MemoryAccounting=yes
> DefaultMemoryLow=0
> DefaultMemoryMin=0
> MemoryMin=0
> MemoryLow=0
> MemoryHigh=infinity
> MemoryMax=infinity
> MemorySwapMax=infinity
> MemoryLimit=infinity
> DevicePolicy=auto
> TasksAccounting=yes
> TasksMax=50833
> IPAccounting=no
> ManagedOOMSwap=auto
> ManagedOOMMemoryPressure=auto
> ManagedOOMMemoryPressureLimit=0
> ManagedOOMPreference=none
> UMask=0022
> LimitCPU=infinity
> LimitCPUSoft=infinity
> LimitFSIZE=infinity
> LimitFSIZESoft=infinity
> LimitDATA=infinity
> LimitDATASoft=infinity
> LimitSTACK=infinity
> LimitSTACKSoft=8388608
> LimitCORE=infinity
> LimitCORESoft=0
> LimitRSS=infinity
> LimitRSSSoft=infinity
> LimitNOFILE=524288
> LimitNOFILESoft=1024
> LimitAS=infinity
> LimitASSoft=infinity
> LimitNPROC=31771
> LimitNPROCSoft=31771
> LimitMEMLOCK=8388608
> LimitMEMLOCKSoft=8388608
> LimitLOCKS=infinity
> LimitLOCKSSoft=infinity
> LimitSIGPENDING=31771
> LimitSIGPENDINGSoft=31771
> LimitMSGQUEUE=819200
> LimitMSGQUEUESoft=819200
> LimitNICE=0
> LimitNICESoft=0
> LimitRTPRIO=0
> LimitRTPRIOSoft=0
> LimitRTTIME=infinity
> LimitRTTIMESoft=infinity
> OOMScoreAdjust=500
> CoredumpFilter=0x23
> Nice=0
> IOSchedulingClass=2
> IOSchedulingPriority=4
> CPUSchedulingPolicy=0
> CPUSchedulingPriority=0
> CPUAffinityFromNUMA=no
> NUMAPolicy=n/a
> TimerSlackNSec=50000
> CPUSchedulingResetOnFork=no
> NonBlocking=no
> StandardInput=null
> StandardOutput=journal
> StandardError=inherit
> TTYReset=no
> TTYVHangup=no
> TTYVTDisallocate=no
> SyslogPriority=30
> SyslogLevelPrefix=yes
> SyslogLevel=6
> SyslogFacility=3
> LogLevelMax=-1
> LogRateLimitIntervalUSec=0
> LogRateLimitBurst=0
> SecureBits=0
> CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf cap_checkpoint_restore
> DynamicUser=no
> RemoveIPC=no
> PrivateTmp=no
> PrivateDevices=no
> ProtectClock=no
> ProtectKernelTunables=no
> ProtectKernelModules=no
> ProtectKernelLogs=no
> ProtectControlGroups=no
> PrivateNetwork=no
> PrivateUsers=no
> PrivateMounts=no
> PrivateIPC=no
> ProtectHome=no
> ProtectSystem=no
> SameProcessGroup=no
> UtmpMode=init
> IgnoreSIGPIPE=yes
> NoNewPrivileges=no
> SystemCallErrorNumber=2147483646
> LockPersonality=no
> RuntimeDirectoryPreserve=no
> RuntimeDirectoryMode=0755
> StateDirectoryMode=0755
> CacheDirectoryMode=0755
> LogsDirectoryMode=0755
> ConfigurationDirectoryMode=0755
> TimeoutCleanUSec=infinity
> MemoryDenyWriteExecute=no
> RestrictRealtime=no
> RestrictSUIDSGID=no
> RestrictNamespaces=no
> MountAPIVFS=no
> KeyringMode=shared
> ProtectProc=default
> ProcSubset=all
> ProtectHostname=no
> KillMode=control-group
> KillSignal=15
> RestartKillSignal=15
> FinalKillSignal=9
> SendSIGKILL=yes
> SendSIGHUP=no
> WatchdogSignal=6
>
>
> *systemctl show cryptsetup.target*
> Id=cryptsetup.target
> Names=cryptsetup.target
> Requires="systemd-cryptsetup at encr\\x2dsdd.service"
> RequiredBy=capture.mount
> WantedBy=sysinit.target
> Before=capture.mount
> After=cryptsetup-pre.target systemd-pcrphase-initrd.service "systemd-cryptsetup at encr\\x2dsdd.service" systemd-ask-password-console.path systemd-ask-password-wall.path
> Description=cryptsetup.target
> LoadState=masked
> ActiveState=inactive
> FreezerState=running
> SubState=dead
> FragmentPath=/etc/systemd/system/cryptsetup.target
> UnitFileState=masked
> UnitFilePreset=enabled
> StateChangeTimestamp=Fri 2025-01-10 09:53:06 UTC
> StateChangeTimestampMonotonic=11420678
> InactiveExitTimestampMonotonic=0
> ActiveEnterTimestampMonotonic=0
> ActiveExitTimestampMonotonic=0
> InactiveEnterTimestampMonotonic=0
> CanStart=no
> CanStop=yes
> CanReload=no
> CanIsolate=no
> CanFreeze=no
> StopWhenUnneeded=no
> RefuseManualStart=no
> RefuseManualStop=no
> AllowIsolate=no
> DefaultDependencies=yes
> OnSuccessJobMode=fail
> OnFailureJobMode=replace
> IgnoreOnIsolate=no
> NeedDaemonReload=no
> JobTimeoutUSec=infinity
> JobRunningTimeoutUSec=infinity
> JobTimeoutAction=none
> ConditionResult=no
> AssertResult=no
> ConditionTimestampMonotonic=0
> AssertTimestampMonotonic=0
> LoadError=org.freedesktop.systemd1.UnitMasked "Unit cryptsetup.target is masked."
> Transient=no
> Perpetual=no
> StartLimitIntervalUSec=10s
> StartLimitBurst=5
> StartLimitAction=none
> FailureAction=none
> SuccessAction=none
> CollectMode=inactive
>
>
> Also, Pasted all the logs in https://gist.github.com/sumitkumar1-trellix/cefb8477e9e6bde05419f8a0e842994d
>
> Please let me know if any other config/logs required.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20250116/3926077d/attachment-0001.htm>