[systemd-devel] systemd unexpectedly dropping into rescue mode - how do I best debug this?
Ingo Molnar
mingo at kernel.org
Thu Oct 4 02:12:08 PDT 2012
hi,
I'm wondering how to debug the following systemd problem: with
recent kernels my Fedora 17 installation drops into rescue mode:
[ OK ] Reached target Sockets.
[ OK ] Reached target Basic System.
Starting Rescue Shell...
[ OK ] Started Rescue Shell.
[ OK ] Reached target Rescue Mode.
Welcome to rescue mode! Type "systemctl default" or ^D to enter default mode.
Type "journalctl" to view system logs. Type "systemctl reboot" to reboot.
I don't know why it did that and none of the messages tell the
user about the reason.
There's no failed units listed:
lyra:~> systemctl --failed
UNIT LOAD ACTIVE SUB JOB DESCRIPTION
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
JOB = Pending job for the unit.
0 units listed. Pass --all to see inactive units, too.
[ btw., a minor usability suggestion: the '0 units listed' line
is the important one. It would be useful to either
color-highlight it or to skip all the preceding info as it's
totally unhelpful to answering the question that the user
asked. ]
Here are the units that are showing some sort of error:
lyra:~> systemctl --all | grep -i err
exim.service error inactive dead exim.service
iscsi.service error inactive dead iscsi.service
iscsid.service error inactive dead iscsid.service
livesys-late.service error inactive dead livesys-late.service
livesys.service error inactive dead livesys.service
named.service error inactive dead named.service
postfix.service error inactive dead postfix.service
remount-rootfs.service error inactive dead remount-rootfs.service
ypserv.service error inactive dead ypserv.service
Here are the kernel parameters:
lyra:~> cat /proc/cmdline
BOOT_IMAGE=/boot/bzImage
root=UUID=dd8873f1-9cfd-4338-b9ec-f4902abdc486 ro single rd.md=0
rd.lvm=0 rd.dm=0 SYSFONT=latarcyrheb-sun16
earlyprintk=serial,ttyS0,115200 console=ttyS0,115200 console=tty
KEYTABLE=us rd.luks=0 LANG=en_US.UTF-8
The systemd version is:
lyra:~> systemctl --version
systemd 44
fedora
+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP
journalctl does not show anything useful:
Logs begin at Thu, 04 Oct 2012 08:13:15 +0200, end at Thu, 04 Oct 2012 11:04:12 +0200.
Oct 04 08:13:15 lyra systemd-journal[463]: Journal started
Oct 04 08:13:15 lyra udevd[458]: specified group 'plugdev' unknown
Oct 04 08:13:16 lyra udevd[458]: unknown key 'BUS' in /etc/udev/rules.d/85-...:3
Oct 04 08:13:16 lyra udevd[458]: invalid rule '/etc/udev/rules.d/85-pcscd_e...3'
Oct 04 08:13:16 lyra udevd[458]: unknown key 'BUS' in /etc/udev/rules.d/85-...:5
Oct 04 08:13:16 lyra udevd[458]: invalid rule '/etc/udev/rules.d/85-pcscd_e...5'
Oct 04 08:13:16 lyra udevd[458]: RUN+="socket:..." support will be removed ...s.
Oct 04 08:13:17 lyra udev-configure-printer[684]: add /bus/pci/drivers/lpc_ich
Oct 04 08:13:17 lyra udev-configure-printer[684]: Failed to get parent
Oct 04 08:13:17 lyra fedora-storage-init[697]: Setting up Logical Volume Man...d
Oct 04 08:13:17 lyra fedora-storage-init[697]: [ OK ]
Oct 04 08:13:17 lyra fedora-storage-init[703]: Setting up Logical Volume Man...d
Oct 04 08:13:17 lyra fedora-storage-init[703]: [ OK ]
Oct 04 08:13:17 lyra lvm[709]: No volume groups found
Oct 04 08:13:18 lyra systemd[1]: Startup finished in 1min 5s 329ms 186us (k...s.
Attached further below is the full dump of the state of all
units. (serial and sshd was started up manually, from the rescue
shell.)
Clues how to debug this further would be welcome.
Thanks,
Ingo
lyra:# systemctl --all
UNIT LOAD ACTIVE SUB JOB DESCRIPTION
proc-sys...misc.automount loaded inactive dead Arbitrary Executable File Formats File System Automount Point
dev-cdrom.device loaded active plugged Slimtype_DVD_A_DS8A2S
dev-disk...0227317.device loaded active plugged Slimtype_DVD_A_DS8A2S
dev-disk...VP1SH3D.device loaded active plugged ST31000528AS
dev-disk...2dpart1.device loaded active plugged ST31000528AS
dev-disk...2dpart2.device loaded active plugged ST31000528AS
dev-disk...b44b89b.device loaded active plugged ST31000528AS
dev-disk...2dpart1.device loaded active plugged ST31000528AS
dev-disk...2dpart2.device loaded active plugged ST31000528AS
dev-disk...93ad3b9.device loaded active plugged ST31000528AS
dev-disk...abdc486.device loaded active plugged ST31000528AS
dev-sda.device loaded active plugged ST31000528AS
dev-sda1.device loaded active plugged ST31000528AS
dev-sda2.device loaded active plugged ST31000528AS
dev-sr0.device loaded active plugged Slimtype_DVD_A_DS8A2S
dev-ttyS0.device loaded active plugged /dev/ttyS0
dev-ttyS1.device loaded active plugged /dev/ttyS1
dev-ttyS2.device loaded active plugged /dev/ttyS2
dev-ttyS3.device loaded active plugged /dev/ttyS3
sys-devi...et-eth0.device loaded active plugged 82576 Gigabit Network Connection
sys-devi...et-eth1.device loaded active plugged 82576 Gigabit Network Connection
sys-devi...da-sda1.device loaded active plugged ST31000528AS
sys-devi...da-sda2.device loaded active plugged ST31000528AS
sys-devi...ock-sda.device loaded active plugged ST31000528AS
sys-devi...ock-sr0.device loaded active plugged Slimtype_DVD_A_DS8A2S
sys-devi...y-ttyS2.device loaded active plugged /sys/devices/platform/serial8250/tty/ttyS2
sys-devi...y-ttyS3.device loaded active plugged /sys/devices/platform/serial8250/tty/ttyS3
sys-devi...y-ttyS0.device loaded active plugged /sys/devices/pnp0/00:05/tty/ttyS0
sys-devi...y-ttyS1.device loaded active plugged /sys/devices/pnp0/00:06/tty/ttyS1
sys-modu...onfigfs.device loaded active plugged /sys/module/configfs
-.mount loaded active mounted /
dev-hugepages.mount loaded active mounted Huge Pages File System
dev-mqueue.mount loaded active mounted POSIX Message Queue File System
media.mount loaded active mounted Media Directory
proc-fs-nfsd.mount loaded inactive dead RPC Pipe File System
proc-sys...fmt_misc.mount loaded inactive dead Arbitrary Executable File Formats File System
sys-fs-f...nections.mount loaded inactive dead FUSE Control File System
sys-kernel-config.mount loaded active mounted Configuration File System
sys-kernel-debug.mount loaded active mounted Debug File System
var-lib-...c_pipefs.mount loaded inactive dead RPC Pipe File System
cups.path loaded inactive dead CUPS Printer Service Spool
systemd-...d-console.path loaded inactive dead Dispatch Password Requests to Console Directory Watch
systemd-...-plymouth.path loaded active waiting Forward Password Requests to Plymouth Directory Watch
systemd-...word-wall.path loaded inactive dead Forward Password Requests to Wall Directory Watch
acpid.service loaded inactive dead ACPI Event Daemon
atd.service loaded inactive dead Job spooling tools
auditd.service loaded inactive dead Security Auditing Service
btseed.service loaded inactive dead SYSV: Start up seed clients for BitTorrent
bttrack.service loaded inactive dead SYSV: Start up tracker for BitTorrent
console-...-start.service loaded active exited Console System Startup Logging
crond.service loaded inactive dead Command Scheduler
cups.service loaded inactive dead CUPS Printing Service
dbus.service loaded active running D-Bus System Message Bus
distccd.service loaded inactive dead Distccd A Distributed Compilation Server
dm-event.service loaded inactive dead Device-mapper event daemon
dracut-shutdown.service loaded inactive dead Restore /run/initramfs
emergency.service loaded inactive dead Emergency Shell
exim.service error inactive dead exim.service
fedora-a...l-mark.service loaded inactive dead Mark the need to relabel after reboot
fedora-a...elabel.service loaded inactive dead Relabel all filesystems, if necessary
fedora-configure.service loaded inactive dead Reconfigure the system on administrator request
fedora-i...-state.service loaded inactive dead Import network configuration from initramfs
fedora-l...odules.service loaded active exited Load legacy module configuration
fedora-readonly.service loaded active exited Configure read-only root support
fedora-s...t-late.service loaded active exited Initialize storage subsystems (RAID, LVM, etc.)
fedora-s...e-init.service loaded active exited Initialize storage subsystems (RAID, LVM, etc.)
fedora-w...torage.service loaded active exited Wait for storage scan
firstboo...phical.service loaded inactive dead firstboot configuration program (graphical mode)
fsck-root.service loaded inactive dead File System Check on Root Device
getty at tty1.service loaded inactive dead Getty on tty1
getty at ttyS0.service loaded inactive dead Getty on ttyS0
gpm.service loaded inactive dead Console Mouse manager
halt.service loaded inactive dead Halt
ip6tables.service loaded inactive dead IPv6 firewall with ip6tables
iptables.service loaded inactive dead IPv4 firewall with iptables
irqbalance.service loaded inactive dead irqbalance daemon
iscsi.service error inactive dead iscsi.service
iscsid.service error inactive dead iscsid.service
killall.service loaded inactive dead killall.service
livesys-late.service error inactive dead livesys-late.service
livesys.service error inactive dead livesys.service
lvm2-monitor.service loaded active exited Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
mcelog.service loaded inactive dead Machine Check Exception Logging Daemon
multipathd.service loaded inactive dead Device-Mapper Multipath Device Controller
named.service error inactive dead named.service
netconsole.service loaded inactive dead SYSV: Initializes network console logging
netfs.service loaded inactive dead LSB: Mount and unmount network filesystems.
network.service loaded inactive dead LSB: Bring up/down networking
NetworkM...online.service loaded inactive dead Network Manager Wait Online
NetworkManager.service loaded inactive dead Network Manager
nfs-idmap.service loaded inactive dead NFSv4 ID-name mapping daemon
nfs-lock.service loaded inactive dead NFS file locking service.
nfs-mountd.service loaded inactive dead NFS Mount Daemon
nfs-rquotad.service loaded inactive dead NFS Remote Quota Server
nfs-secure.service loaded inactive dead Secure NFS
nfs-server.service loaded inactive dead NFS Server
pcscd.service loaded inactive dead PC/SC Smart Card Daemon
plymouth...t-wait.service loaded inactive dead Wait for Plymouth Boot Screen to Quit
plymouth-quit.service loaded inactive dead Terminate Plymouth Boot Screen
plymouth...-write.service loaded inactive dead Tell Plymouth To Write Out Runtime Data
plymouth-start.service loaded inactive dead Show Plymouth Boot Screen
postfix.service error inactive dead postfix.service
poweroff.service loaded inactive dead Power-Off
prefdm.service loaded inactive dead Display Manager
quotacheck.service loaded active exited File System Quota Check
quotaon.service loaded active exited Enable File System Quotas
rc-local.service loaded inactive dead /etc/rc.d/rc.local Compatibility
reboot.service loaded inactive dead Reboot
remount-rootfs.service error inactive dead remount-rootfs.service
rescue.service loaded active running Rescue Shell
rpcbind.service loaded inactive dead RPC bind service
rsyslog.service loaded inactive dead System Logging Service
sendmail.service loaded inactive dead Sendmail Mail Transport Agent
serial-g... at ttyS0.service loaded active running Serial Getty on ttyS0
sm-client.service loaded inactive dead Sendmail Mail Transport Client
smartd.service loaded inactive dead Self Monitoring and Reporting Technology (SMART) Daemon
sshd.service loaded active running OpenSSH server daemon
sssd.service loaded inactive dead System Security Services Daemon
systemd-...onsole.service loaded inactive dead Dispatch Password Requests to Console
systemd-...ymouth.service loaded inactive dead Forward Password Requests to Plymouth
systemd-...d-wall.service loaded inactive dead Forward Password Requests to Wall
systemd-binfmt.service loaded inactive dead Set Up Additional Binary Formats
systemd-initctl.service loaded inactive dead /dev/initctl Compatibility Daemon
systemd-journald.service loaded active running Journal Service
systemd-logind.service loaded active running Login Service
systemd-...s-load.service loaded inactive dead Load Kernel Modules
systemd-...d-load.service loaded inactive dead Load Random Seed
systemd-...d-save.service loaded inactive dead Save Random Seed
systemd-...ollect.service loaded inactive dead Collect Read-Ahead Data
systemd-...d-done.service loaded inactive dead Stop Read-Ahead Data Collection
systemd-...replay.service loaded inactive dead Replay Read-Ahead Data
systemd-...unt-fs.service loaded active exited Remount Root and Kernel File Systems
systemd-shutdownd.service loaded inactive dead Delayed Shutdown Service
systemd-sysctl.service loaded active exited Apply Kernel Variables
systemd-...-clean.service loaded inactive dead Cleanup of Temporary Directories
systemd-...-setup.service loaded active exited Recreate Volatile Files and Directories
systemd-...nlevel.service loaded inactive dead Update UTMP about System Runlevel Changes
systemd-...utdown.service loaded inactive dead Update UTMP about System Shutdown
systemd-...ssions.service loaded inactive dead Permit User Sessions
systemd-...-setup.service loaded active exited Setup Virtual Console
tcsd.service loaded inactive dead LSB: Init script for TCSD
udev-settle.service loaded active exited udev Wait for Complete Device Initialization
udev-trigger.service loaded active exited udev Coldplug all Devices
udev.service loaded active running udev Kernel Device Manager
ypbind.service loaded inactive dead NIS/YP (Network Information Service) Clients to NIS Domain Binder
ypserv.service error inactive dead ypserv.service
cups.socket loaded active listening CUPS Printing Service Sockets
dbus.socket loaded active running D-Bus System Message Bus Socket
dm-event.socket loaded active listening Device-mapper event daemon FIFOs
pcscd.socket loaded active listening PC/SC Smart Card Daemon Activation Socket
rpcbind.socket loaded active listening RPCbind Server Activation Socket
syslog.socket loaded inactive dead Syslog Socket
systemd-initctl.socket loaded active listening /dev/initctl Compatibility Named Pipe
systemd-journald.socket loaded active running Journal Socket
systemd-shutdownd.socket loaded active listening Delayed Shutdown Socket
udev-control.socket loaded active running udev Control Socket
udev-kernel.socket loaded active running udev Kernel Socket
dev-disk...\x2dpart2.swap loaded active active /dev/disk/by-id/ata-ST31000528AS_6VP1SH3D-part2
dev-disk...\x2dpart2.swap loaded active active /dev/disk/by-id/wwn-0x5000c5001b44b89b-part2
dev-disk...5d93ad3b9.swap loaded active active /dev/disk/by-uuid/01d112f8-9e5a-4d5f-9506-b4f5d93ad3b9
dev-sda2.swap loaded inactive dead /dev/sda2
basic.target loaded active active Basic System
cryptsetup.target loaded active active Encrypted Volumes
emergency.target loaded inactive dead Emergency Mode
final.target loaded inactive dead Final Step
getty.target loaded inactive dead Login Prompts
graphical.target loaded inactive dead Graphical Interface
local-fs-pre.target loaded active active Local File Systems (Pre)
local-fs.target loaded active active Local File Systems
multi-user.target loaded inactive dead Multi-User
network.target loaded inactive dead Network
nss-lookup.target loaded inactive dead Host and Network Name Lookups
nss-user-lookup.target loaded inactive dead User and Group Name Lookups
remote-fs-pre.target loaded inactive dead Remote File Systems (Pre)
remote-fs.target loaded inactive dead Remote File Systems
rescue.target loaded active active Rescue Mode
shutdown.target loaded inactive dead Shutdown
sockets.target loaded active active Sockets
swap.target loaded active active Swap
sysinit.target loaded active active System Initialization
syslog.target loaded inactive dead Syslog
umount.target loaded inactive dead Unmount All Filesystems
systemd-...ead-done.timer loaded inactive dead Stop Read-Ahead Data Collection 10s After Completed Startup
systemd-...es-clean.timer loaded active waiting Daily Cleanup of Temporary Directories
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
JOB = Pending job for the unit.
184 units listed.
More information about the systemd-devel
mailing list