[systemd-devel] Confusing journal information - journal size

[Anne Mulhern]

----- Original Message -----
> From: "David Sommerseth" <davids at redhat.com>
> To: "Anne Mulhern" <amulhern at redhat.com>
> Cc: systemd-devel at lists.freedesktop.org
> Sent: Monday, July 20, 2015 11:29:22 AM
> Subject: Re: [systemd-devel] Confusing journal information - journal size
> 
> On 20/07/15 15:31, Anne Mulhern wrote:
> > ----- Original Message -----
> >> From: "Colin Guthrie" <gmane at colin.guthr.ie>
> >> To: systemd-devel at lists.freedesktop.org
> >> Sent: Monday, July 20, 2015 5:38:38 AM
> >> Subject: Re: [systemd-devel] Confusing journal information - journal size
> >>
> >> David Sommerseth wrote on 17/07/15 14:28:
> >>> On 17/07/15 13:31, Mantas Mikulėnas wrote:
> >>>> On Fri, Jul 17, 2015 at 2:13 PM, David Sommerseth <davids at redhat.com
> >>>> <mailto:davids at redhat.com>> wrote:
> >>>>
> >>>>
> >>>>     Hi,
> >>>>
> >>>>     I'm looking through some journals now, and even though I've seen it
> >>>>     a
> >>>>     few times I haven't thought about it until now.
> >>>>
> >>>>        systemd-journal[1151]: Runtime journal is using 8.0M (max allowed
> >>>>                  4.0G, trying to leave 4.0G free of 63.7G available →
> >>>>                  current limit 4.0G).
> >>>>
> >>>>     Could this line be cleaned up so you don't have to look up a man
> >>>>     page
> >>>>     to
> >>>>     try to figure out what this really means?  Here's my uneducated
> >>>>     guess
> >>>>     and confusion of this line:
> >>>>
> >>>>     * Runtime journal is using 8.0M
> >>>>       - Okay, so currently the journal uses 8MB of disk-space.  No
> >>>>       problem.
> >>>>
> >>>>     * max allowed 4.0G
> >>>>       - Okay, so the journal should not grow beyond 4GB, makes sense.
> >>>>       No
> >>>>         problem.
> >>>>
> >>>>     * trying to leave 4.0G free of 63.7G available
> >>>>       - Uhm, what!? So it will grow until there is 4GB left on the
> >>>>         filesystem?  Not so okay.
> >>>>
> >>>>
> >>>> It chooses the /smallest/ limit, not largest. (Common sense...) For
> >>>> example, if you had only 5 GB space available, the journal would not
> >>>> grow beyond 1 GB.
> >>>>  
> >>>>
> >>>>     * current limit 4.0G
> >>>>       - Ehh ... okay ... so make up your mind, please!  So will the
> >>>>         journal grow until 4GB or 59.7GB.
> >>>>
> >>>>
> >>>> This *is* it making up its mind: "min(limit 1, limit 2) → resulting
> >>>> limit"
> >>>>
> >>>>     But then I looked into /var/log/journal ...
> >>>>
> >>>>       # du --si -s /var/log/journal/
> >>>>       4.3G  /var/log/journal/
> >>>>
> >>>>     I do see that both system,journal and user-UID.journal are both
> >>>>     8.4MB,
> >>>>     and from that I can guess what the log entry tried to tell me with
> >>>>     "Runtime journal" ... but how is /that/ information useful for me,
> >>>>     from
> >>>>     a sys-admin point of view?
> >>>>
> >>>>
> >>>> "Runtime" here means /run, as opposed to persistent in /var. They have
> >>>> separately configurable limits, since /run is in RAM and /var is usually
> >>>> on disk. (Though, I'm not entirely sure what purpose the runtime journal
> >>>> even serves, when /var is available.)
> >>>
> >>> Fair enough.  But you are missing my point.
> >>>
> >>> How this information is presented do require some detail knowledge of
> >>> the journal.  Don't think like a developer who have poked at the journal
> >>> code.  Think like a sys-admin who looks through the logs looking for
> >>> issues.  Then you want to have the answer straight in your face, not
> >>> needing to go elsewhere to read about these things.  In fact most admins
> >>> will probably have forgotten what they were going to look for when they
> >>> move their eyes of the log data.
> >>>
> >>> If it is considered important information, fine.  But present it in a
> >>> far more understandable way for those who just uses the journal.  Right
> >>> now, I'm not surprised if most sys-admins read that line as useless
> >>> gibberish - "Yeah, yeah, journal will waste some space on my drive".
> >>
> >> Yeah, I can't disagree with David. Not sure how best to tidy it up, but
> >> some rework would definitely be nice.
> >>
> >> Col
> >>
> > 
> > After seeing the explanation, the best complete and correct (AFAICT)
> > formulation I could come up with was,
> > 
> > "Runtime journal is using 8.0M (max allowed = min(4.0G, S s.t. total
> > memory(63.7 G) - S = 4.0 G (59.7 G), available memory (16.2 G)) = 4.0G)"
> > 
> > which is compelled to use math speak for clarity and succinctness.
> > 
> > Dunno how happy most sys-admins would be with that.
> > 
> > - mulhern
> 
> But is all that information really needed?
> 
> If I try to see this from a sys-admin point of view there are two
> numbers which are important to me: 1) Current state 2) Final journal
> limit size.  From how I see it, how the journal code ends up with a
> certain number is only useful when you're developing/debugging the
> journal.  Remember: Less is more.
> 
> 
> --
> kind regards,
> 
> David Sommerseth
> 

That seems reasonable to me.

And

"Runtime journal is using 8.0M (max allowed = 4.0G)"

is telling it like it is (for this system).

The additional info really does seem like its appropriate
situation is in debugging output.

Also, now that I look at the source it seems like my formulation was wrong,
journal is claiming 
that current limit is s->cached_available_space + sum, which isn't
really the minimum of any of the other values mentioned.
So something more is going on.

- mulhern