[systemd-devel] systemd unexpectedly dropping into rescue mode - how do I best debug this?

Ingo Molnar mingo at kernel.org
Thu Oct 4 02:12:08 PDT 2012


hi,

I'm wondering how to debug the following systemd problem: with 
recent kernels my Fedora 17 installation drops into rescue mode:

 [  OK  ] Reached target Sockets.
 [  OK  ] Reached target Basic System.
          Starting Rescue Shell...
 [  OK  ] Started Rescue Shell.
 [  OK  ] Reached target Rescue Mode.
 Welcome to rescue mode! Type "systemctl default" or ^D to enter default mode.
 Type "journalctl" to view system logs. Type "systemctl reboot" to reboot.

I don't know why it did that and none of the messages tell the 
user about the reason.

There's no failed units listed:

 lyra:~> systemctl --failed
 UNIT LOAD   ACTIVE SUB JOB DESCRIPTION

 LOAD   = Reflects whether the unit definition was properly loaded.
 ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
 SUB    = The low-level unit activation state, values depend on unit type.
 JOB    = Pending job for the unit.

 0 units listed. Pass --all to see inactive units, too.

[ btw., a minor usability suggestion: the '0 units listed' line 
  is the important one. It would be useful to either 
  color-highlight it or to skip all the preceding info as it's 
  totally unhelpful to answering the question that the user 
  asked. ]

Here are the units that are showing some sort of error:

lyra:~> systemctl --all | grep -i err
exim.service              error  inactive dead          exim.service
iscsi.service             error  inactive dead          iscsi.service
iscsid.service            error  inactive dead          iscsid.service
livesys-late.service      error  inactive dead          livesys-late.service
livesys.service           error  inactive dead          livesys.service
named.service             error  inactive dead          named.service
postfix.service           error  inactive dead          postfix.service
remount-rootfs.service    error  inactive dead          remount-rootfs.service
ypserv.service            error  inactive dead          ypserv.service

Here are the kernel parameters:

 lyra:~> cat /proc/cmdline 

 BOOT_IMAGE=/boot/bzImage 
 root=UUID=dd8873f1-9cfd-4338-b9ec-f4902abdc486 ro single rd.md=0 
 rd.lvm=0 rd.dm=0 SYSFONT=latarcyrheb-sun16 
 earlyprintk=serial,ttyS0,115200 console=ttyS0,115200 console=tty 
 KEYTABLE=us rd.luks=0 LANG=en_US.UTF-8

The systemd version is:

 lyra:~> systemctl --version
 systemd 44
 fedora
 +PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP

journalctl does not show anything useful:

Logs begin at Thu, 04 Oct 2012 08:13:15 +0200, end at Thu, 04 Oct 2012 11:04:12 +0200.
Oct 04 08:13:15 lyra systemd-journal[463]: Journal started
Oct 04 08:13:15 lyra udevd[458]: specified group 'plugdev' unknown
Oct 04 08:13:16 lyra udevd[458]: unknown key 'BUS' in /etc/udev/rules.d/85-...:3
Oct 04 08:13:16 lyra udevd[458]: invalid rule '/etc/udev/rules.d/85-pcscd_e...3'
Oct 04 08:13:16 lyra udevd[458]: unknown key 'BUS' in /etc/udev/rules.d/85-...:5
Oct 04 08:13:16 lyra udevd[458]: invalid rule '/etc/udev/rules.d/85-pcscd_e...5'
Oct 04 08:13:16 lyra udevd[458]: RUN+="socket:..." support will be removed ...s.
Oct 04 08:13:17 lyra udev-configure-printer[684]: add /bus/pci/drivers/lpc_ich
Oct 04 08:13:17 lyra udev-configure-printer[684]: Failed to get parent
Oct 04 08:13:17 lyra fedora-storage-init[697]: Setting up Logical Volume Man...d
Oct 04 08:13:17 lyra fedora-storage-init[697]: [  OK  ]
Oct 04 08:13:17 lyra fedora-storage-init[703]: Setting up Logical Volume Man...d
Oct 04 08:13:17 lyra fedora-storage-init[703]: [  OK  ]
Oct 04 08:13:17 lyra lvm[709]: No volume groups found
Oct 04 08:13:18 lyra systemd[1]: Startup finished in 1min 5s 329ms 186us (k...s.

Attached further below is the full dump of the state of all 
units. (serial and sshd was started up manually, from the rescue 
shell.)

Clues how to debug this further would be welcome.

Thanks,

	Ingo

lyra:# systemctl --all

UNIT                      LOAD   ACTIVE   SUB       JOB DESCRIPTION
proc-sys...misc.automount loaded inactive dead          Arbitrary Executable File Formats File System Automount Point
dev-cdrom.device          loaded active   plugged       Slimtype_DVD_A_DS8A2S
dev-disk...0227317.device loaded active   plugged       Slimtype_DVD_A_DS8A2S
dev-disk...VP1SH3D.device loaded active   plugged       ST31000528AS
dev-disk...2dpart1.device loaded active   plugged       ST31000528AS
dev-disk...2dpart2.device loaded active   plugged       ST31000528AS
dev-disk...b44b89b.device loaded active   plugged       ST31000528AS
dev-disk...2dpart1.device loaded active   plugged       ST31000528AS
dev-disk...2dpart2.device loaded active   plugged       ST31000528AS
dev-disk...93ad3b9.device loaded active   plugged       ST31000528AS
dev-disk...abdc486.device loaded active   plugged       ST31000528AS
dev-sda.device            loaded active   plugged       ST31000528AS
dev-sda1.device           loaded active   plugged       ST31000528AS
dev-sda2.device           loaded active   plugged       ST31000528AS
dev-sr0.device            loaded active   plugged       Slimtype_DVD_A_DS8A2S
dev-ttyS0.device          loaded active   plugged       /dev/ttyS0
dev-ttyS1.device          loaded active   plugged       /dev/ttyS1
dev-ttyS2.device          loaded active   plugged       /dev/ttyS2
dev-ttyS3.device          loaded active   plugged       /dev/ttyS3
sys-devi...et-eth0.device loaded active   plugged       82576 Gigabit Network Connection
sys-devi...et-eth1.device loaded active   plugged       82576 Gigabit Network Connection
sys-devi...da-sda1.device loaded active   plugged       ST31000528AS
sys-devi...da-sda2.device loaded active   plugged       ST31000528AS
sys-devi...ock-sda.device loaded active   plugged       ST31000528AS
sys-devi...ock-sr0.device loaded active   plugged       Slimtype_DVD_A_DS8A2S
sys-devi...y-ttyS2.device loaded active   plugged       /sys/devices/platform/serial8250/tty/ttyS2
sys-devi...y-ttyS3.device loaded active   plugged       /sys/devices/platform/serial8250/tty/ttyS3
sys-devi...y-ttyS0.device loaded active   plugged       /sys/devices/pnp0/00:05/tty/ttyS0
sys-devi...y-ttyS1.device loaded active   plugged       /sys/devices/pnp0/00:06/tty/ttyS1
sys-modu...onfigfs.device loaded active   plugged       /sys/module/configfs
-.mount                   loaded active   mounted       /
dev-hugepages.mount       loaded active   mounted       Huge Pages File System
dev-mqueue.mount          loaded active   mounted       POSIX Message Queue File System
media.mount               loaded active   mounted       Media Directory
proc-fs-nfsd.mount        loaded inactive dead          RPC Pipe File System
proc-sys...fmt_misc.mount loaded inactive dead          Arbitrary Executable File Formats File System
sys-fs-f...nections.mount loaded inactive dead          FUSE Control File System
sys-kernel-config.mount   loaded active   mounted       Configuration File System
sys-kernel-debug.mount    loaded active   mounted       Debug File System
var-lib-...c_pipefs.mount loaded inactive dead          RPC Pipe File System
cups.path                 loaded inactive dead          CUPS Printer Service Spool
systemd-...d-console.path loaded inactive dead          Dispatch Password Requests to Console Directory Watch
systemd-...-plymouth.path loaded active   waiting       Forward Password Requests to Plymouth Directory Watch
systemd-...word-wall.path loaded inactive dead          Forward Password Requests to Wall Directory Watch
acpid.service             loaded inactive dead          ACPI Event Daemon
atd.service               loaded inactive dead          Job spooling tools
auditd.service            loaded inactive dead          Security Auditing Service
btseed.service            loaded inactive dead          SYSV: Start up seed clients for BitTorrent
bttrack.service           loaded inactive dead          SYSV: Start up tracker for BitTorrent
console-...-start.service loaded active   exited        Console System Startup Logging
crond.service             loaded inactive dead          Command Scheduler
cups.service              loaded inactive dead          CUPS Printing Service
dbus.service              loaded active   running       D-Bus System Message Bus
distccd.service           loaded inactive dead          Distccd A Distributed Compilation Server
dm-event.service          loaded inactive dead          Device-mapper event daemon
dracut-shutdown.service   loaded inactive dead          Restore /run/initramfs
emergency.service         loaded inactive dead          Emergency Shell
exim.service              error  inactive dead          exim.service
fedora-a...l-mark.service loaded inactive dead          Mark the need to relabel after reboot
fedora-a...elabel.service loaded inactive dead          Relabel all filesystems, if necessary
fedora-configure.service  loaded inactive dead          Reconfigure the system on administrator request
fedora-i...-state.service loaded inactive dead          Import network configuration from initramfs
fedora-l...odules.service loaded active   exited        Load legacy module configuration
fedora-readonly.service   loaded active   exited        Configure read-only root support
fedora-s...t-late.service loaded active   exited        Initialize storage subsystems (RAID, LVM, etc.)
fedora-s...e-init.service loaded active   exited        Initialize storage subsystems (RAID, LVM, etc.)
fedora-w...torage.service loaded active   exited        Wait for storage scan
firstboo...phical.service loaded inactive dead          firstboot configuration program (graphical mode)
fsck-root.service         loaded inactive dead          File System Check on Root Device
getty at tty1.service        loaded inactive dead          Getty on tty1
getty at ttyS0.service       loaded inactive dead          Getty on ttyS0
gpm.service               loaded inactive dead          Console Mouse manager
halt.service              loaded inactive dead          Halt
ip6tables.service         loaded inactive dead          IPv6 firewall with ip6tables
iptables.service          loaded inactive dead          IPv4 firewall with iptables
irqbalance.service        loaded inactive dead          irqbalance daemon
iscsi.service             error  inactive dead          iscsi.service
iscsid.service            error  inactive dead          iscsid.service
killall.service           loaded inactive dead          killall.service
livesys-late.service      error  inactive dead          livesys-late.service
livesys.service           error  inactive dead          livesys.service
lvm2-monitor.service      loaded active   exited        Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
mcelog.service            loaded inactive dead          Machine Check Exception Logging Daemon
multipathd.service        loaded inactive dead          Device-Mapper Multipath Device Controller
named.service             error  inactive dead          named.service
netconsole.service        loaded inactive dead          SYSV: Initializes network console logging
netfs.service             loaded inactive dead          LSB: Mount and unmount network filesystems.
network.service           loaded inactive dead          LSB: Bring up/down networking
NetworkM...online.service loaded inactive dead          Network Manager Wait Online
NetworkManager.service    loaded inactive dead          Network Manager
nfs-idmap.service         loaded inactive dead          NFSv4 ID-name mapping daemon
nfs-lock.service          loaded inactive dead          NFS file locking service.
nfs-mountd.service        loaded inactive dead          NFS Mount Daemon
nfs-rquotad.service       loaded inactive dead          NFS Remote Quota Server
nfs-secure.service        loaded inactive dead          Secure NFS
nfs-server.service        loaded inactive dead          NFS Server
pcscd.service             loaded inactive dead          PC/SC Smart Card Daemon
plymouth...t-wait.service loaded inactive dead          Wait for Plymouth Boot Screen to Quit
plymouth-quit.service     loaded inactive dead          Terminate Plymouth Boot Screen
plymouth...-write.service loaded inactive dead          Tell Plymouth To Write Out Runtime Data
plymouth-start.service    loaded inactive dead          Show Plymouth Boot Screen
postfix.service           error  inactive dead          postfix.service
poweroff.service          loaded inactive dead          Power-Off
prefdm.service            loaded inactive dead          Display Manager
quotacheck.service        loaded active   exited        File System Quota Check
quotaon.service           loaded active   exited        Enable File System Quotas
rc-local.service          loaded inactive dead          /etc/rc.d/rc.local Compatibility
reboot.service            loaded inactive dead          Reboot
remount-rootfs.service    error  inactive dead          remount-rootfs.service
rescue.service            loaded active   running       Rescue Shell
rpcbind.service           loaded inactive dead          RPC bind service
rsyslog.service           loaded inactive dead          System Logging Service
sendmail.service          loaded inactive dead          Sendmail Mail Transport Agent
serial-g... at ttyS0.service loaded active   running       Serial Getty on ttyS0
sm-client.service         loaded inactive dead          Sendmail Mail Transport Client
smartd.service            loaded inactive dead          Self Monitoring and Reporting Technology (SMART) Daemon
sshd.service              loaded active   running       OpenSSH server daemon
sssd.service              loaded inactive dead          System Security Services Daemon
systemd-...onsole.service loaded inactive dead          Dispatch Password Requests to Console
systemd-...ymouth.service loaded inactive dead          Forward Password Requests to Plymouth
systemd-...d-wall.service loaded inactive dead          Forward Password Requests to Wall
systemd-binfmt.service    loaded inactive dead          Set Up Additional Binary Formats
systemd-initctl.service   loaded inactive dead          /dev/initctl Compatibility Daemon
systemd-journald.service  loaded active   running       Journal Service
systemd-logind.service    loaded active   running       Login Service
systemd-...s-load.service loaded inactive dead          Load Kernel Modules
systemd-...d-load.service loaded inactive dead          Load Random Seed
systemd-...d-save.service loaded inactive dead          Save Random Seed
systemd-...ollect.service loaded inactive dead          Collect Read-Ahead Data
systemd-...d-done.service loaded inactive dead          Stop Read-Ahead Data Collection
systemd-...replay.service loaded inactive dead          Replay Read-Ahead Data
systemd-...unt-fs.service loaded active   exited        Remount Root and Kernel File Systems
systemd-shutdownd.service loaded inactive dead          Delayed Shutdown Service
systemd-sysctl.service    loaded active   exited        Apply Kernel Variables
systemd-...-clean.service loaded inactive dead          Cleanup of Temporary Directories
systemd-...-setup.service loaded active   exited        Recreate Volatile Files and Directories
systemd-...nlevel.service loaded inactive dead          Update UTMP about System Runlevel Changes
systemd-...utdown.service loaded inactive dead          Update UTMP about System Shutdown
systemd-...ssions.service loaded inactive dead          Permit User Sessions
systemd-...-setup.service loaded active   exited        Setup Virtual Console
tcsd.service              loaded inactive dead          LSB: Init script for TCSD
udev-settle.service       loaded active   exited        udev Wait for Complete Device Initialization
udev-trigger.service      loaded active   exited        udev Coldplug all Devices
udev.service              loaded active   running       udev Kernel Device Manager
ypbind.service            loaded inactive dead          NIS/YP (Network Information Service) Clients to NIS Domain Binder
ypserv.service            error  inactive dead          ypserv.service
cups.socket               loaded active   listening     CUPS Printing Service Sockets
dbus.socket               loaded active   running       D-Bus System Message Bus Socket
dm-event.socket           loaded active   listening     Device-mapper event daemon FIFOs
pcscd.socket              loaded active   listening     PC/SC Smart Card Daemon Activation Socket
rpcbind.socket            loaded active   listening     RPCbind Server Activation Socket
syslog.socket             loaded inactive dead          Syslog Socket
systemd-initctl.socket    loaded active   listening     /dev/initctl Compatibility Named Pipe
systemd-journald.socket   loaded active   running       Journal Socket
systemd-shutdownd.socket  loaded active   listening     Delayed Shutdown Socket
udev-control.socket       loaded active   running       udev Control Socket
udev-kernel.socket        loaded active   running       udev Kernel Socket
dev-disk...\x2dpart2.swap loaded active   active        /dev/disk/by-id/ata-ST31000528AS_6VP1SH3D-part2
dev-disk...\x2dpart2.swap loaded active   active        /dev/disk/by-id/wwn-0x5000c5001b44b89b-part2
dev-disk...5d93ad3b9.swap loaded active   active        /dev/disk/by-uuid/01d112f8-9e5a-4d5f-9506-b4f5d93ad3b9
dev-sda2.swap             loaded inactive dead          /dev/sda2
basic.target              loaded active   active        Basic System
cryptsetup.target         loaded active   active        Encrypted Volumes
emergency.target          loaded inactive dead          Emergency Mode
final.target              loaded inactive dead          Final Step
getty.target              loaded inactive dead          Login Prompts
graphical.target          loaded inactive dead          Graphical Interface
local-fs-pre.target       loaded active   active        Local File Systems (Pre)
local-fs.target           loaded active   active        Local File Systems
multi-user.target         loaded inactive dead          Multi-User
network.target            loaded inactive dead          Network
nss-lookup.target         loaded inactive dead          Host and Network Name Lookups
nss-user-lookup.target    loaded inactive dead          User and Group Name Lookups
remote-fs-pre.target      loaded inactive dead          Remote File Systems (Pre)
remote-fs.target          loaded inactive dead          Remote File Systems
rescue.target             loaded active   active        Rescue Mode
shutdown.target           loaded inactive dead          Shutdown
sockets.target            loaded active   active        Sockets
swap.target               loaded active   active        Swap
sysinit.target            loaded active   active        System Initialization
syslog.target             loaded inactive dead          Syslog
umount.target             loaded inactive dead          Unmount All Filesystems
systemd-...ead-done.timer loaded inactive dead          Stop Read-Ahead Data Collection 10s After Completed Startup
systemd-...es-clean.timer loaded active   waiting       Daily Cleanup of Temporary Directories

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
JOB    = Pending job for the unit.

184 units listed.


More information about the systemd-devel mailing list