[systemd-devel] Help! iSCSI based file systems with "_netdev" causing ordering cycles to occur (random services and mounts fail)

[Lennart Poettering]

On Do, 26.10.23 19:03, Tony Rodriguez (unixpro1970 at gmail.com) wrote:

> Experiencing this same issue with iSCSI and systemd-239 for RH8/Rocky8 and
> RH9/Rocky9 system-252. Nothing was done on my end to create this issue.  In
> other words, no custom mount/unit files or services, just your typical ISO
> install and rpm updates.
>
> An ordering cycle occurs, when "_netdev" is specified within /etc/fstab for
> systemd.  This happens with systemd-239-14 and systemd-239-18 using iSCSI
> based file systems.    Seems others are experiencing this as well (see link
> below).  I can also confirm this happens with systemd-252 (RH9/Rocky9)l.
> Especially if "_netdev" is used with either "/var" or "/usr" iSCSI based
> devices/file systems.  The system may not boot, may not mount file systems,
> may not start services/unit files, and the system becomes slow during system
> boot.
>
> Does anyone know of a fix/patch and root cause for this?
>
> Please see this link:
> https://issues.redhat.com/browse/RHEL-12987?jql=project%20%3D%20RHEL%20AND%20affectedVersion%20%3D%20rhel-9.2.0%20AND%20text%20~%20%22iscsi%22
>
> # cat /etc/fstab
> [...]
> /dev/mapper/rhel-root /                       xfs defaults,_netdev 0 0
> UUID=2177a7fc-bc41-43e4-bdc1-d231a5eb4680 /boot xfs defaults,_netdev 0 0
> /dev/mapper/rhel-tmp /tmp                    xfs defaults,_netdev 0 0
> /dev/mapper/rhel-var /var                    xfs
> defaults,_netdev,x-initrd.mount 0 0
> /dev/mapper/rhel-var_log /var/log                xfs defaults,_netdev 0 0
> /dev/mapper/rhel-var_tmp /var/tmp                xfs defaults,_netdev 0 0
>
> # journalctl -b | grep deleted
> Oct 13 08:15:35 vm-isci8 systemd[1]: basic.target: Job tmp.mount/start
> deleted to break ordering cycle starting with basic.target/start
> Oct 13 08:15:35 vm-isci8 systemd[1]: network.target: Job
> network-pre.target/start deleted to break ordering cycle starting with
> network.target/start
> Oct 13 08:15:35 vm-isci8 systemd[1]: NetworkManager.service: Job
> dbus.socket/start deleted to break ordering cycle starting with
> NetworkManager.service/start

/tmp must be available during early boot already, and your
NetworkManager service is apparently a late boot service. Hence you
have a cycle: you want that /tmp/ is mounted after the network, but
your network is configured really late. But /tmp is necessary during
early boot. BOOM!

Two ways out:

1. Don't make /tmp an iscsi mount. Bad idea anyway. Just use tmpfs for
   it, like everyone else.

2. Upgrade to a better network management solution that has no
   problems with running in early boot, for example systemd-networkd.

Lennart

--
Lennart Poettering, Berlin