[systemd-devel] Can coredumpctl work independent of journald?

[Lennart Poettering]

On Thu, 12.05.16 08:15, P.R.Dinesh (pr.dinesh at gmail.com) wrote:

> Thank you Lennart,
> I would like to explain my system scenario.
> 
> We are using systemd version 219. (Updating to 229 is in progress).
> 
> Configured for persistent storage for both Journal and Coredump (Coredump
> is stored externallly)
> 
> The logs and coredump are stored in another partition
> "/var/diagnostics/logs" and "/var/diagnostics/coredump".  We do symbolic
> link between the /var/lib/systemd/coredump and logs to those folders.
> 
> Coredump and journald is configured to utilized upto 10% of the disk
> space(total disk space is ~400MB) which would allocate 40MB to journal logs
> and 40MB to coredump.  For some reason (under investigation) some of our
> daemons are generating too much logs which makes the journald to reach the
> 40MB limit within 6 hours.  Hence journald starts wrapping around.
> Meanwhile some daemons have also crashed and core dumped.
> 
> Now when I do coredumplist, none of those coredumps are shown.

I see.

> Also I tried launching the coredumpctl with the coredump file both using
>  the pid name as well as using the coredump file name.  Since we dont have
> the journal entry coredumpctl is not launching them,  can we atleast have
> the coredumpctl launch the gdb using the core dump file name?

The coredumps are simply compressed, use the xz tools to decompress
them. Note however, that the xz file format generated by the xz
libraries was incomptible with the xz tool for a long time, which is
why the xz support in the journal and the coredumping code was
experimental until v229. Make sure to upgrade to v229 if you want to
decompress the coredumps with the normal "unxz". 

> 
> [prd at localhost ~]$ coredumpctl gdb
> core.lshw.1000.9bb41758bba94306b39e751048e0cee9.23993.1462871523000000.xz
> No match found.
> [prd at localhost ~]$ coredumpctl gdb 23993
> No match found.

If you have v229 this should work:

    unxz < /var/lib/systemd/coredump/core.lshw.1000.9bb41758bba94306b39e751048e0cee9.23993.1462871523000000.xz > ~/coredump
    gdb ~/coredump
    …

> In summary, the frequency of logs are higher and the frequency of core
> dumps are very less in our system which leads to the loss of coredump
> information.
> 
> I am thinking of two solutions here
> 1) Enhance coredumpctl to launch the gdb using the coredump file name
> 2) Store the Journal logs for coredump seperately from other journal logs
> so that they could be maintained for long duration (is this
> feasible?)

A feature that has been requested before is that we add
priority-sensitive rotation to the journal: keep error messages with
levels of ERROR or higher around for longer than INFO and lower or
so. Given that the coredumps are logged at CRIT level, this would
cover your case nicely. 

However, that's a feature request only, and I think it would make
sense to have, but nobody hacked that up yet.

(along with this feature we should probably also add support for
storing low-priority messages in /run only, so that debug stuff is
kept around only during runtime, but not after)

Lennart

-- 
Lennart Poettering, Red Hat