[systemd-devel] How to debug systemd services failing to start with 11/SEGV?

[Alexander Dahl]

Hello Lennart,

thanks for your quick reply, see below.

Am Tue, Apr 09, 2024 at 03:53:24PM +0200 schrieb Lennart Poettering:
> On Di, 09.04.24 14:42, Alexander Dahl (ada at thorsis.com) wrote:
> 
> > Hello everyone,
> >
> > I'm currently trying to build a firmware for an embedded device and
> > running into trouble because systemd seems to crash.  The BSP is
> > based on pengutronix DistroKit (master) built with ptxdist and the
> > target is the Microchip SAM9X60-Curiosity board, which is arm v5te
> > architecture (that board is not part of DistroKit, support for that is
> > in an upper layer of mine not public yet (?)).
> >
> > Everything is quite recent, building systemd version 255.2 currently.
> > On startup I get messages like this (this is the first one, later on
> > there are lot more, all with the same status):
> >
> >    [   11.175650] systemd[1]: systemd-journald.service: Main process exited, code=killed, status=11/SEGV
> >    [   11.239679] systemd[1]: systemd-journald.service: Failed with result 'signal'.
> >    [   11.292640] systemd[1]: Failed to start systemd-journald.service.
> >    [FAILED] Failed to start systemd-journald.service.
> >    See 'systemctl status systemd-journald.service' for details.
> >
> > The system drops me on a shell later, where I can run the above
> > mentioned command, which gives:
> >
> >     ~ # systemctl status systemd-journald.service
> >     x systemd-journald.service - Journal Service
> >          Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static)
> >          Active: failed (Result: signal) since Tue 2024-04-09 11:44:52 UTC; 11min a>
> >     TriggeredBy: x systemd-journald-dev-log.socket
> >                  * systemd-journald-audit.socket
> >                  x systemd-journald.socket
> >            Docs: man:systemd-journald.service(8)
> >                  man:journald.conf(5)
> >         Process: 197 ExecStart=/usr/lib/systemd/systemd-journald (code=killed, sign>
> >        Main PID: 197 (code=killed, signal=SEGV)
> >        FD Store: 0 (limit: 4224)
> >             CPU: 330ms
> >
> > This does not help me much.  Other services crashing: systemd-udevd
> > and systemd-timesyncd, also with status 11/SEGV which is segmentation
> > fault, right?
> 
> Yes.
> 
> > I had this board running with an older version of systemd, but I can
> > not remember which was the last good version.
> >
> > Could anyone give me a hint please how to debug this?
> 
> "coredumpctl gdb" should get open the most recent backtrace for you.

This gives:

    ~ # coredumpctl gdb
    No journal files were found.
    No match found.

> The coredump should also show up in the logs with a backtrace.

I only have serial console output.  journald is crashing.  With
`dmesg` I see systemd messages in kernel log, but no backtrace.
gdbserver is installed on target, no gdb currently.  Trying to get a
coredump tomorrow.

Greets
Alex