[systemd-devel] /proc and /sys get unmounted during boot from NFS, which results in boot error

Tue Feb 3 14:00:57 PST 2015

Dear systemd-devel list,

I'm trying to debug the following problem:

For some unknown reason, /proc and /sys get unmounted during boot
from a NFS mounted rootfs. Booting to an emergency shell, I can observe
them disappear by first calling mount (which reads from /proc due to /etc/mtab 
being a symlink) and then calling "ls /proc". After calling ls, mount obviously
complains about not being able to read mtab, instead of listing the mounts as 
before.

A 2nd or 3rd call of mount before listing /proc works fine, too.

This happens on an gentoo-amd64 system using kernel 3.18.5 and systemd-218.

Booting with debug options yields no further information, no new
messages appear while the debug shell is running and the messages
before are not very interesting (as far as I can tell) [--> end of mail].

Booting the very same installation with the same initramfs from a disk works fine, 
even when forcing the disk to be mounted read-only, as the NFS share is.

About a year ago, I installed a number-crunching cluster, whose nodes are running
from a read-only NFS share. This was also a gentoo based system running systemd-215. 
Stuff works fine, there. Thus, I tried version 215 and even 216 for the new 
installation, too, yet without success. 

What might cause systemd to unmount /proc and /sys? Or might this issue not be 
related to systemd at all?

Thanks for your input!

Olaf

#############

[....] [leading kernel messages skipped]
[   17.857082] systemd[1]: Mounting cgroup to /sys/fs/cgroup/cpu,cpuacct of type cgroup with options cpu,cpuacct.
[   17.857308] systemd[1]: Mounting cgroup to /sys/fs/cgroup/memory of type cgroup with options memory.
[   17.857505] systemd[1]: Mounting cgroup to /sys/fs/cgroup/devices of type cgroup with options devices.
[   17.857694] systemd[1]: Mounting cgroup to /sys/fs/cgroup/net_cls of type cgroup with options net_cls.
[   17.857882] systemd[1]: Mounting cgroup to /sys/fs/cgroup/cpuset of type cgroup with options cpuset.
[   17.858073] systemd[1]: Mounting cgroup to /sys/fs/cgroup/freezer of type cgroup with options freezer.
[   17.858258] systemd[1]: Mounting cgroup to /sys/fs/cgroup/blkio of type cgroup with options blkio.
[   17.858479] systemd[1]: systemd 218 running in system mode. (+PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK -SYSVINIT +UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN)
[   17.860362] systemd[1]: Detected architecture 'x86-64'.
[   17.879066] systemd[1]: Using cgroup controller name=systemd. File system hierarchy is at /sys/fs/cgroup/systemd.
[   17.879273] systemd[1]: Installed release agent.
[   17.879434] systemd[1]: Set up TFD_TIMER_CANCEL_ON_SET timerfd.
[   17.882492] systemd[160]: Spawned /usr/lib/systemd/system-generators/systemd-debug-generator as 161.
[   17.882782] systemd[160]: Spawned /usr/lib/systemd/system-generators/systemd-gpt-auto-generator as 162.
[   17.883095] systemd[160]: Spawned /usr/lib/systemd/system-generators/systemd-getty-generator as 163.
[   17.883403] systemd[160]: Spawned /usr/lib/systemd/system-generators/systemd-system-update-generator as 164.
[   17.883685] systemd[160]: Spawned /usr/lib/systemd/system-generators/gentoo-local-generator as 165.
[   17.883969] systemd[160]: Spawned /usr/lib/systemd/system-generators/systemd-efi-boot-generator as 166.
[   17.884273] systemd[160]: Spawned /usr/lib/systemd/system-generators/systemd-fstab-generator as 167.
[   17.884565] systemd[160]: Spawned /usr/lib/systemd/system-generators/systemd-hibernate-resume-generator as 168.
[   17.923410] systemd-efi-boot-generator[166]: Not an EFI boot, exiting.
[   17.923445] systemd-fstab-generator[167]: Parsing /etc/fstab
[   17.925772] systemd-fstab-generator[167]: Found entry what=134.96.30.183:/exports/raid/client/distribution/gentoo-amd64 where=/ type=nfs nofail=no noauto=no
[   17.926259] systemd-gpt-auto-generator[162]: Not a EFI boot, not creating root mount.
[   17.987813] systemd-gpt-auto-generator[162]: Root file system not on a (single) block device.
[   17.987848] systemd-fstab-generator[167]: Found entry what=home-srv:/home where=/home type=nfs nofail=yes noauto=no
[   17.988250] systemd[160]: /usr/lib/systemd/system-generators/systemd-gpt-auto-generator succeeded.
[   17.988422] systemd[160]: /usr/lib/systemd/system-generators/systemd-hibernate-resume-generator succeeded.
[   17.988582] systemd[160]: /usr/lib/systemd/system-generators/systemd-debug-generator succeeded.
[   17.988745] systemd[160]: /usr/lib/systemd/system-generators/systemd-getty-generator succeeded.
[   17.988900] systemd[160]: /usr/lib/systemd/system-generators/systemd-system-update-generator succeeded.
[   18.090371] systemd[160]: /usr/lib/systemd/system-generators/gentoo-local-generator succeeded.
[   18.090528] systemd[160]: /usr/lib/systemd/system-generators/systemd-fstab-generator succeeded.
[   18.090680] systemd[160]: /usr/lib/systemd/system-generators/systemd-efi-boot-generator succeeded.
[   18.090976] systemd[1]: /usr/lib/systemd/system-generators succeeded.
[   18.091723] systemd[1]: Looking for unit files in (higher priority first):
[   18.091833] systemd[1]: 	/etc/systemd/system
[   18.091931] systemd[1]: 	/run/systemd/system
[   18.092029] systemd[1]: 	/run/systemd/generator
[   18.092128] systemd[1]: 	/usr/local/lib/systemd/system
[   18.092230] systemd[1]: 	/usr/lib64/systemd/system
[   18.092330] systemd[1]: 	/lib/systemd/system
[   18.092428] systemd[1]: SysV init scripts and rcN.d links support disabled
[   18.129877] systemd[1]: Failed to load configuration for plymouth-start.service: No such file or directory
[   18.135713] systemd[1]: Failed to load configuration for syslog.service: No such file or directory
[   18.159901] systemd[1]: Failed to load configuration for auditd.service: No such file or directory
[   18.186044] systemd[1]: Using notification socket /run/systemd/notify
[   18.186166] systemd[1]: Successfully created private D-Bus server.
[   18.186275] systemd[1]: tmp.mount changed dead -> mounted
[   18.186397] systemd[1]: var.mount changed dead -> mounted
[   18.186507] systemd[1]: -.mount changed dead -> mounted
[   18.186619] systemd[1]: Activating default unit: emergency.target
[   18.186727] systemd[1]: Trying to enqueue job emergency.target/start/isolate
[   18.186882] systemd[1]: Installed new job emergency.service/start as 2
[   18.187000] systemd[1]: Installed new job -.slice/start as 5
[   18.187105] systemd[1]: Installed new job emergency.target/start as 1
[   18.187213] systemd[1]: Installed new job system.slice/start as 4
[   18.187324] systemd[1]: Enqueued job emergency.target/start as 1
[   18.187431] systemd[1]: Loaded units and determined initial transaction in 305ms.
[   18.187610] systemd[1]: Child 159 (switch_root) died (code=exited, status=0/SUCCESS)
[   18.187818] systemd[1]: Received SIGCHLD from PID 160 (n/a).
[   18.187935] systemd[1]: Starting Root Slice.
[   18.188210] systemd[1]: -.slice changed dead -> active
[   18.188314] systemd[1]: Job -.slice/start finished, result=done
[   18.188445] systemd[1]: Created slice Root Slice.
[   18.188555] systemd[1]: Starting System Slice.
[   18.188740] systemd[1]: system.slice changed dead -> active
[   18.188845] systemd[1]: Job system.slice/start finished, result=done
[   18.188958] systemd[1]: Created slice System Slice.
[   18.189066] systemd[1]: Starting Emergency Shell...
[   18.189337] systemd[1]: About to execute: /bin/plymouth quit
[   18.189562] systemd[1]: Forked /bin/plymouth as 170
[   18.189725] systemd[1]: emergency.service changed dead -> start-pre
[   18.189834] systemd[1]: Job emergency.service/start finished, result=done
[   18.189954] systemd[1]: Started Emergency Shell.
[   18.190078] systemd[1]: Starting Emergency Mode.
[   18.190178] systemd[1]: emergency.target changed dead -> active
[   18.190291] systemd[1]: Job emergency.target/start finished, result=done
[   18.190406] systemd[1]: Reached target Emergency Mode.
[   18.190526] systemd[1]: Startup finished in 17.677s (kernel) + 352ms (userspace) = 18.030s.
[   18.191601] systemd[170]: Executing: /bin/plymouth quit
[   18.191996] systemd[170]: Failed at step EXEC spawning /bin/plymouth: No such file or directory
[   18.192354] systemd[1]: Received SIGCHLD from PID 170 ((plymouth)).
[   18.192479] systemd[1]: Child 170 ((plymouth)) died (code=exited, status=203/EXEC)
[   18.192645] systemd[1]: Child 170 belongs to emergency.service
[   18.192754] systemd[1]: emergency.service: control process exited, code=exited status=203
[   18.192959] systemd[1]: emergency.service running next control command for state start-pre
[   18.193145] systemd[1]: About to execute: /bin/echo -e 'Welcome to emergency mode! After logging in, type "journalctl -xb" to view\nsystem logs, "systemctl reboot" to reboot, "systemctl default" or ^D to\ntry again to boot into default mode.'
[   18.193525] systemd[1]: Forked /bin/echo as 172
[   18.193998] systemd[172]: Executing: /bin/echo -e 'Welcome to emergency mode! After logging in, type "journalctl -xb" to view\nsystem logs, "systemctl reboot" to reboot, "systemctl default" or ^D to\ntry again to boot into default mode.'
[   18.224074] systemd[1]: Accepted new private connection.
[   18.224228] systemd-cgroups-agent[171]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/agent interface=org.freedesktop.systemd1.Agent member=Released cookie=1 reply_cookie=0 error=n/a
[   18.224963] systemd[1]: Accepted new private connection.
[   18.225097] systemd[1]: Got message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/agent interface=org.freedesktop.systemd1.Agent member=Released cookie=1 reply_cookie=0 error=n/a
[   18.225113] systemd-cgroups-agent[173]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/agent interface=org.freedesktop.systemd1.Agent member=Released cookie=1 reply_cookie=0 error=n/a
[   18.225654] systemd[1]: Got disconnect on private connection.
[   18.226067] systemd[1]: Got message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/agent interface=org.freedesktop.systemd1.Agent member=Released cookie=1 reply_cookie=0 error=n/a
[   18.226357] systemd[1]: Got disconnect on private connection.
[   18.359352] systemd[1]: Received SIGCHLD from PID 172 (echo).
[   18.359477] systemd[1]: Child 172 (echo) died (code=exited, status=0/SUCCESS)
[   18.359609] systemd[1]: Child 172 belongs to emergency.service
[   18.359719] systemd[1]: emergency.service: control process exited, code=exited status=0
[   18.359938] systemd[1]: emergency.service got final SIGCHLD for state start-pre
[   18.360112] systemd[1]: About to execute: /bin/sh -c '/sbin/sulogin; /usr/bin/systemctl --fail --no-block default'
[   18.360376] systemd[1]: Forked /bin/sh as 176
[   18.360596] systemd[1]: emergency.service changed start-pre -> running
[   18.360817] systemd[176]: Executing: /bin/sh -c '/sbin/sulogin; /usr/bin/systemctl --fail --no-block default'
[   18.405949] systemd[1]: Accepted new private connection.
[   18.406092] systemd-cgroups-agent[175]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/agent interface=org.freedesktop.systemd1.Agent member=Released cookie=1 reply_cookie=0 error=n/a
[   18.406816] systemd[1]: Accepted new private connection.
[   18.406943] systemd[1]: Got message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/agent interface=org.freedesktop.systemd1.Agent member=Released cookie=1 reply_cookie=0 error=n/a
[   18.406944] systemd-cgroups-agent[178]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/agent interface=org.freedesktop.systemd1.Agent member=Released cookie=1 reply_cookie=0 error=n/a
[   18.407478] systemd[1]: Got disconnect on private connection.
[   18.407883] systemd[1]: Got message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/agent interface=org.freedesktop.systemd1.Agent member=Released cookie=1 reply_cookie=0 error=n/a
[   18.408145] systemd[1]: Got disconnect on private connection