[systemd-devel] systemd-nspawn container not starting on RHEL9.0

[Thomas Archambault]

Good day everyone on the dev list,
We are adding an analysis tool to our application that uses the host's 
rootfs as one of its inputs.

As a proof of concept, we used systemd-nspawn on Fedora 34 to create an 
isolated container environment using the host's rootfs as the 
container's rootfs and things worked correctly and as expected. The 
host's rootfs is analyzed with tmp and results files generated within 
the container without persistent modifications affecting the host's 
rootfs. Since RHEL is our ultimate target platform, I've been trying to 
duplicate our work over RHEL9.0 without success with the container not 
being instantiated.

I've tried to boil down the duplication code to the simplest example, 
which is also an example in the man page $ sudo systemd-nspawn -xbD/. As 
with my prototyping, the container does not seem to be instantiated.
Any help with troubleshooting, or specific known issues, or requests for 
more data would be appreciated.

TIA
tparchambault
ps: Regarding security - selinux is in Permissive mode. I do not know if 
seccomp filters are getting in the way or not; This is an out-ot-the-box 
RHEL9.0 base workstation install. In the FC34 prototype, I did need to 
allow certain syscalls via --system-call-filter in order to get a daemon 
within the container to run correctly but afaik that should have no 
bearing on the instantiation of the container.


==== On a RHEL9.0 host bash session ====

[toma at localhost ~]$ systemctl --version
systemd 250 (250-6.el9_0)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS 
+OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD 
+LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 
+XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT 
default-hierarchy=unified

[toma at localhost ~]$ uname -a
Linux localhost.localdomain 5.14.0-70.17.1.el9_0.x86_64 #1 SMP PREEMPT 
Tue Jun 14 11:32:10 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
[toma at localhost ~]$

[toma at localhost ~]$ sudo time systemd-nspawn -D / -xb
^C^C^C^C^CCommand terminated by signal 15
40.81user 298.75system 6:29.72elapsed 87%CPU (0avgtext+0avgdata 
8524maxresident)k
205032inputs+0outputs (0major+3287minor)pagefaults 0swaps
[toma at localhost ~]$

==== In another bash session on the same host ====
[toma at localhost ~]$ sudo machinectl list
[sudo] password for toma:
No machines.
[toma at localhost ~]$ sudo pkill nspawn
[toma at localhost ~]$

== In the original host bash session, w/increased logging and strace 
capture ==

[toma at localhost ~]$ sudo SYSTEMD_LOG_LEVEL=debug strace -o 
Development/nspawn.strace.rhel90.out systemd-nspawn -D / -xb
[sudo] password for toma:
Setting RLIMIT_CPU to infinity.
Setting RLIMIT_FSIZE to infinity.
Setting RLIMIT_DATA to infinity.
Setting RLIMIT_STACK to 8388608:infinity.
Setting RLIMIT_CORE to 0:infinity.
Setting RLIMIT_RSS to infinity.
Setting RLIMIT_NPROC to 14657.
Setting RLIMIT_NOFILE to 1024:524288.
Setting RLIMIT_MEMLOCK to 65536.
Setting RLIMIT_AS to infinity.
Setting RLIMIT_LOCKS to infinity.
Setting RLIMIT_SIGPENDING to 14657.
Setting RLIMIT_MSGQUEUE to 819200.
Setting RLIMIT_NICE to 0.
Setting RLIMIT_RTPRIO to 0.
Setting RLIMIT_RTTIME to infinity.
Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
Terminated
[toma at localhost ~]$

As with the first run, killed via pkill from the other terminal session.

Fwiw, on Fedora 34, the log debug output shows the instantiation of the
container after the "Found csgroup2..." line, with the container working as
documented eventually presenting the login prompt, i.e.

...
Setting RLIMIT_RTTIME to infinity.
Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
Spawning container fedora-1aabc34e0a52a82b on /.#machine.6e49b8aa974c6f37.
Press ^] three times within 1s to kill container.
Outer child is initializing.
Mounting / (MS_REC|MS_SLAVE "")...
...

[  OK  ] Finished Update UTMP about System Runlevel Changes.

Fedora 34 (Workstation Edition)
Kernel 5.11.12-300.fc34.x86_64 on an x86_64 (console)

fedora-1aabc34e0a52a82b login:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20220803/359f5243/attachment.htm>