[systemd-devel] journald leaking maps?

[Michael Olbrich]

On Fri, Oct 12, 2012 at 03:58:06PM +0100, Colin Guthrie wrote:
> 'Twas brillig, and Michael Olbrich at 12/10/12 09:52 did gyre and gimble:
> > On Fri, Oct 12, 2012 at 04:31:46AM -0400, Dave Reisner wrote:
> >> On Fri, Oct 12, 2012 at 09:19:08AM +0100, Colin Guthrie wrote:
> >>> 'Twas brillig, and David Strauss at 12/10/12 05:39 did gyre and gimble:
> >>>> On Thu, Oct 11, 2012 at 3:31 PM, Colin Guthrie <gmane at colin.guthr.ie> wrote:
> >>>>> Something is obviously not good there! journald is using something in
> >>>>> the region of 250MB res.
> >>>>>
> >>>>> What's the best way to debug this?
> >>>>
> >>>> What version are you on? The Fedora 17 journal does excessive mapping
> >>>> that's fixed in current versions.
> >>>
> >>> Arg, I forgot that detail - I'm on 194, no patches to journal related
> >>> stuff applied.
> >>>
> >>> As others are not seeing it jump out at them, I figured it might be due
> >>> to just using /run and not flushing it to disk etc.
> >>>
> > 
> > Well the mapped memory is the same physical memory, that is used in the
> > tmpfs. You'll probably run out of memory there first, before journald runs
> > out of virtual address space. And that will happen, even if journald does
> > not map it. Setting RuntimeMaxUse in journald.conf should help here.
> 
> Well perhaps, but with systemd 189 (I think, perhaps a bit earlier) I
> certainly didn't see this kind of memory usage. I'm certainly hitting
> OOMs much more regularly now unless I restart systemd occasionally.
> 
> Also, that explanation doesn't really explain:
> 
> 
> [root at jimmy ~]# cat /proc/$(pidof systemd-journald)/status | grep Vm
> VmPeak:	 6611116 kB
> VmSize:	 6611112 kB

I've tried to flood the journal with messages for a while now. VmPeak is
stable and VmSize grows until it reaches VmPeak and the drops
significantly.
This looks like some limit is reached and then journald releases most of
its memory.

> VmLck:	       0 kB
> VmPin:	       0 kB
> VmHWM:	  253692 kB
> VmRSS:	  227680 kB
> VmData:	     960 kB
> VmStk:	     136 kB
> VmExe:	     164 kB
> VmLib:	    2804 kB
> VmPTE:	   12876 kB
> VmSwap:	     172 kB
> 
> [root at jimmy ~]# cat /proc/$(pidof systemd-journald)/maps | grep -c /run/log/
> 1939

This goes up and down with VmSize for me.

> [root at jimmy ~]# du -sh /run/log/
> 10M	/run/log/
> 
> 
> So, I've got about 2k mmaps, using about 230Mb RSS and yet my journal
> data is only 10M in tmpfs... doesn't sound right to me!
> 
> Unless all this RSS memory is just fake and pretend and just different
> maps onto that 10MB and it's really "free", but my understanding was
> that it was the VmSize that was all the virtual mmap stuff, not the
> VmRSS... please correct me if I'm wrong.

VmSize is, as far as I know the currently used virtual memory. This does
not mean that there is any real memory associated with it. VmRSS is the
total actual memory that is mapped into the address space of the
application.
Usually mapping a lot of files is no problem. When the memory is needed
then the kernel just frees some pages. After all, it can just read the
files from disk again when needed.
On a tmpfs however thats not possible, there is no persistent storage
backing the files.

What I think happens for you is the following:
For some reason, journald is not releasing old mappings. It still rotates
files in /run/log/journal/, so "du -sh /run/log/" seems correct. However,
the old mappings mean, that even though the file is deleted, its content is
still there. So the mappings are for files in /run/log/journal/, they just
don't show up because there is no directory entry for them any more.

I guess the algorithm that decides when to release old mappings is broken
for you.

Regards,
Michael


-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |