[systemd-devel] systemd unexpectedly dropping into rescue mode - how do I best debug this?

[Ingo Molnar]

* Ingo Molnar <mingo at kernel.org> wrote:

> > Could you add:
> >   systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M
> > to the kernel command line? It will print all userspace log to the
> > kernel buffer, which is the most reliable way to store the logs when
> > stuff goes wrong that early during bootup.
> > 
> > It should tell us more what's going on, and why you end up in the rescue shell.
> 
> Sure, will try this straight away.

Just tried that and unfortunately I ran into the debugging 
problem category outlined in:

  https://bugzilla.redhat.com/show_bug.cgi?id=861912

I'm physically away from the test system in question, and the 
serial output stops with:

[...]
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
         Starting Rescue Shell...
[  OK  ] Started Rescue Shell.
[  OK  ] Reached target Rescue Mode.
[   72.072313] systemd[1]: Startup finished in 1min 5s 192ms 343us (kernel) + 2s 27ms 776us (initrd) + 

No rescue shell was spawned on the serial console.

The above is all the info I have - shouldn't 
systemd.log_level=debug have produced more data, to the serial 
console (which is the kernel console as well)?

Btw., this kind of complication is why the following debugging 
concept:

> I'm just trying to figure out how to analyze this best without 
> having to reboot the system.

would have been nice - rebooting both destroys debug data and 
it's sometimes hard to reach a usable debugging shell on a 
failing system.

I'll have physical console access to the system later today.

Thanks,

	Ingo