[systemd-devel] Can coredumpctl work independent of journald?

[P.R.Dinesh]

Additionally can we improve the journal log rotate system to keep the
higher severity messages (like coredump) for longer duration and remove
lower priority messages to get spaces.

On Thu, May 12, 2016 at 8:15 AM, P.R.Dinesh <pr.dinesh at gmail.com> wrote:

> Thank you Lennart,
> I would like to explain my system scenario.
>
> We are using systemd version 219. (Updating to 229 is in progress).
>
> Configured for persistent storage for both Journal and Coredump (Coredump
> is stored externallly)
>
> The logs and coredump are stored in another partition
> "/var/diagnostics/logs" and "/var/diagnostics/coredump".  We do symbolic
> link between the /var/lib/systemd/coredump and logs to those folders.
>
> Coredump and journald is configured to utilized upto 10% of the disk
> space(total disk space is ~400MB) which would allocate 40MB to journal logs
> and 40MB to coredump.  For some reason (under investigation) some of our
> daemons are generating too much logs which makes the journald to reach the
> 40MB limit within 6 hours.  Hence journald starts wrapping around.
> Meanwhile some daemons have also crashed and core dumped.
>
> Now when I do coredumplist, none of those coredumps are shown.
>
> Also I tried launching the coredumpctl with the coredump file both using
>  the pid name as well as using the coredump file name.  Since we dont have
> the journal entry coredumpctl is not launching them,  can we atleast have
> the coredumpctl launch the gdb using the core dump file name?
>
> [prd at localhost ~]$ coredumpctl gdb
> core.lshw.1000.9bb41758bba94306b39e751048e0cee9.23993.1462871523000000.xz
> No match found.
> [prd at localhost ~]$ coredumpctl gdb 23993
> No match found.
>
>
> In summary, the frequency of logs are higher and the frequency of core
> dumps are very less in our system which leads to the loss of coredump
> information.
>
> I am thinking of two solutions here
> 1) Enhance coredumpctl to launch the gdb using the coredump file name
> 2) Store the Journal logs for coredump seperately from other journal logs
> so that they could be maintained for long duration (is this feasible?)
>
> Thank you
> Regards,
> Dinesh
>
> On Wed, May 11, 2016 at 10:25 PM, Lennart Poettering <
> lennart at poettering.net> wrote:
>
>> On Wed, 11.05.16 20:31, P.R.Dinesh (pr.dinesh at gmail.com) wrote:
>>
>> > I have set the journald to be persistant and limit its size to 40MB.
>> > I had a process coredumped and the coredump file is found in
>> > /var/log/systemd/coredump
>> >
>> > When I run coredumpctl this coredump is not shown.
>> >
>> > Later I found that the core dump log is missing from the Journal ( the
>> > journal got wrapped since it reached the size limitation).
>> >
>> > I think coredumpctl depends on journal to display the coredump.  Can't
>> it
>> > search for the coredump files present in the coredump folder and list
>> those
>> > files?
>>
>> We use the metadata and the filtering the journal provides us with,
>> and the coredump on disk is really just secondary, external data to
>> that, that can be lifecycled quicker than the logging data. We extract
>> the backtrace from the coredump at the momemt the coredump happens,
>> and all that along with numerous metadata fields is stored in the
>> journal. In fact storing the coredump is optional, because in many
>> setups the short backtrace in the logs is good enough, and the
>> coredump is less important.
>>
>> So, generally the concept here really is that logs are cheap, and thus
>> you keep around more of them; and coredumps are large and thus you
>> lifecycle them quicker. If I understand correctly what you want is the
>> opposite: you want a quicker lifecycle for the logs but keep the
>> coredumps around for longer. I must say, I am not entirely sure where
>> such a setup would be a good idea though... i.e. wanting persistent
>> coredumps but volatile logging sounds a strange combination to
>> me... Can you make a good case for this?
>>
>> But yeah, we really don't cover what you are asking for right now, and
>> I am not sure we should...
>>
>> > Also can I launch the coredumpctl gdb by providing a compressed core
>> > file.
>>
>> If you configure systemd-coredump to store the coredumps compressed
>> (which is in fact the default), then "coredumpctl gdb" will implicitly
>> decompress them so that gdb can do its work.
>>
>> Lennart
>>
>> --
>> Lennart Poettering, Red Hat
>>
>
>
>
> --
> With Kind Regards,
> Dinesh P Ramakrishnan
>



-- 
With Kind Regards,
Dinesh P Ramakrishnan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20160512/ef9f5170/attachment-0001.html>