[systemd-devel] PID 1 "crashes" during boot, disabling systemd-coredump, but system boots normally anyway

[Lennart Poettering]

xOn Fr, 21.02.25 20:39, Joanne Norman (joanne.norman at spectralogic.com) wrote:

> Linux distribution: Poky reference distribution from Yocto Project release 5.0 "Scarthgap"
> systemd version: 255.4
> kernel: 6.6.25-intel-pk-standard
>
> Systemd is disabling coredumps during bootup, claiming that it's
> "Due to PID 1 having crashed". However, aside from that log line, I
> can find no evidence that PID 1 actually crashed, and the system
> continues to boot normally. Log lines immediately afterward indicate
> that udevd has crashed at the same time, but again, it appears to
> recover, as the system boots normally afterward and "systemd[1]:"
> log statements continue to appear.
>
> root at intel-corei7-64:~# journalctl -b | grep coredump -C10
> -unrelated context elided-
> Feb 21 19:00:56 intel-corei7-64 systemd-coredump[396]: Due to PID 1 having crashed coredump collection will now be turned off.
> Feb 21 19:00:56 intel-corei7-64 systemd-coredump[396]: Resource limits disable core dumping for process 103 (udevd).
> Feb 21 19:00:56 intel-corei7-64 systemd-coredump[396]: Process 103 (udevd) of user 0 terminated abnormally without generating a coredump.
> Feb 21 19:00:57 intel-corei7-64 kernel: igb 0000:01:00.0 enp1s0: igb: enp1s0 NIC Link is Up 100 Mbps Half Duplex, Flow Control: None
> Feb 21 19:00:57 intel-corei7-64 kernel: igb 0000:01:00.0: EEE Disabled: unsupported at half duplex. Re-enable using ethtool when at full duplex.
> -unrelated context elided-
>
> root at intel-corei7-64:~# cat /proc/sys/kernel/core_pattern
> |/bin/false
>
> Setting LogLevel=debug in /etc/systemd/system.conf and rebooting
> produces a hell of a lot of extra logs, but nothing that mentions
> PID 1 crashing or provides any additional details about the udevd
> crash. Same for setting udev_log=debug in
> /etc/udev/udev.conf. coredumpctl lists the udev crash, but makes no
> mention of the purported "PID 1" crash (note, this log is from a
> separate boot, so the PID and times are different from above):
>
> root at intel-corei7-64:~# coredumpctl info 102
>            PID: 102 (udevd)
>            UID: 0 (root)
>            GID: 0 (root)
>         Signal: 6 (ABRT)
>      Timestamp: Thu 2025-02-20 22:29:28 UTC (30min ago)
>   Command Line: udevd --daemon
>     Executable: /usr/bin/udevadm
> Control Group: /init.scope
>           Unit: init.scope
>          Slice: -.slice
>        Boot ID: cc6df2a288af4b3586aa28ddf229ff5e
>     Machine ID: 911fe70f4be4456e9b23f5857dbd73a2
>       Hostname: intel-corei7-64
>        Storage: none
>        Message: Process 102 (udevd) of user 0 terminated abnormally without generating a coredump.
> root at intel-corei7-64:~#
>
> At this point I'm not certain how to continue diagnosing this
> seemingly-spurious "PID 1 crash", since debug logging produced no
> leads and there's no coredump for me to look at for context. Any
> advice, further troubleshooting steps I can take, or pointers toward
> more information on what might actually be going on would be greatly
> appreciated.

Please contact your distribution about this. My educated guess is that
they maintain their own private initrd implementation that uses a
forked version of udev (because this is about an udev binary called
just "udevd", but that's not how it is called upstream. We call it
"systemd-udevd"). They apparently did a bad job at forking it though,
since it hit SIGABRT. (I am not sure what the point of renaming it is
though, except for increasing churn? i mean, if you think it's worth
patching through our sources to drop the "systemd-" prefix, you
probably are in the systemd hater camp, because otherwise why would
you? but why in heaven do you then use it to start systemd after all?
your distro seems to have some split personality issues there)

And apparently they left the process running over the initrd→host
transition, inside the top-level cgroup (which causes it to be
migrated into the /init.scope cgroup by systemd early on), and be
considered a process auxiliary to PID 1.

Anyway, there's a lot wrong with your distro here, please contact
them. This doesn't appear to be an upstream systemd issue at all, …

Lennart

--
Lennart Poettering, Berlin