[systemd-devel] How to use cgroups for Tracker?

[Martyn Russell]

On 21/10/14 12:21, Lennart Poettering wrote:
> On Tue, 21.10.14 12:03, Martyn Russell (martyn at lanedo.com) wrote:
>
>>> What precisely are you setting with sched_setscheulder() and ioprio_set()?
>>
>> https://git.gnome.org/browse/tracker/tree/src/libtracker-common/tracker-sched.c#n29
>>
>> and
>>
>> https://git.gnome.org/browse/tracker/tree/src/libtracker-common/tracker-ioprio.c#n131
>
> Looks OK.
>
>>>> is much better, but some people really want Tracker to
>>>>    a) only perform when they're away from their computer OR
>>>>    b) be completely unnoticeable.
>>>>
>>>> Now, we can do a) without cgroups, but I believe b) would be better done
>>>> using cgroups.
>>>
>>> Well, it will always be noticable since it generates IO. It will
>>> always show up in top/iotop hence. If you want it to not interfere
>>> with other things beind done on the system use
>>> ioprio_set/sched_setscheduler to mark things as batch jobs.
>>
>> You're right. When we get reports of 100% CPU use I am not so concerned if
>> the system is not doing anything else - but I suspect what leads people to
>> find out about the 100% situation is related to their UI not
>> responding.
>
> Hmm. I would always have assumed that tracker is strictly IO-bound,
> not CPU-bound, hence 100% sounds suspicious to me. What precisely is
> tracker doing there that it needs to crunch that much data? Just
> extracting some meta-data from a set of files doesn't sound like
> something CPU intensive.

Tracker does quite a number of things that could require some level of 
processing power.

Just some off the top of my head:

- Parsing words any any language in large quantities of text
- Unaccenting words
- Unicode normalization
- Case folding
- Stemming

and more.

It also depends on which process or binary you're talking about, but 
extractors (like the one using poppler for PDFs) can easily require a 
LOT of processing power to handle complex PDFs. We only care about the 
text usually, but that's not always under our control unless we write 
our own extractor.

On top of all this, you have a SPARQL engine which translates to SQL and 
then has to perform some DB operation - which adds to the processing 
requirement.

It might be worth measuring again to see where this processing power is 
going.

>>>>> Why don't the kernel API calls pointed out above not suffice?
>>>>
>>>> I think it depends on the API and who you talk to.
>>>>
>>>> For me, the APIs work quite well. However, we still get bug reports. I find
>>>> this quite hard to quantify personally because the filesystems, hardware and
>>>> version of Tracker all come into play and can make quite some
>>>> difference.
>>>
>>> What precisely are the bug reports about?
>>
>> Often 100% CPU use.
>> Sometimes high memory use (but that's mostly related to tracker-extract).
>> Disk use occasionally when someone is indexing a crazy data set.
>>
>> The latest bug report:
>> https://bugzilla.gnome.org/show_bug.cgi?id=676713
>
> Well, looking at that bug it appears to me that this is caused because
> you try to use inotify for something it shouldn't be used for: to
> recursively watch an entire directory subtree. If you fake recursive
> fs watching by recursively adding all subdirs to the inotify watch
> then of course you eat a lot of CPU.

In our experience, watching a tree and seeing changes in that tree 
through inotify is not the expensive part (unless you're currency is 
FDs). It does depend on what operations are taking place.

The bug report above is certainly asking for trouble :) but it's just an 
example.

> The appropriate fix for this is to not make use of inotify this way,
> which might mean fixing the kernel to provide recursive subscription
> to fs events for unpriviliged processes. Sorry if that's
> disappointing, but no amount of cgriups can work around that.

Not at all. Actually, I really would like something like that in the 
kernel and user space has been asking for a while :)

> Don't try to work around limitations of kernel APIs by implementing
> inherently not scalabale algorithms in userspace. I mean, you
> implemented something that scales O(n) with n the numbers of
> dirs. That's what you need to fix, there's no way around that. Just
> looking for magic wands in cgroups and scheduling facilities to make
> an algorithm that fundamentally scales badly acceptable is not going
> to work.

OK.

Could I ask one more favour from you Lennart, could you possibly reply 
on the bug report where your fellow RedHat-ers :) suggest using cgroups?

https://bugzilla.gnome.org/show_bug.cgi?id=737663#c6

>>>> The one API that doesn't really work for us is setrlimit(), mainly because
>>>> we have to guess the memory threshold (which we never get right) and we get
>>>> a lot of SIGABRTs that get reported as bugs. I suppose we could catch
>>>> SIGABRT and exit gracefully, but lately, we've agreed (as a team) that if an
>>>> extractor or library we depend on uses 2Gb of memory and brings a smaller
>>>> system to its knees, it's a bug and we should just fix the
>>>> extractor/library, not try to compensate for it. Sadly, there are always
>>>> these sort of bugs and it's precisely why tracker-extract is a separate
>>>> process.
>>>
>>> What functionality of setrlimit() are you precisely using?
>>
>> https://git.gnome.org/browse/tracker/tree/src/libtracker-common/tracker-os-dependant-unix.c?h=tracker-1.2#n289
>
> This is misleading, as RLIMIT_AS and RLIMIT_DATA limit address space,
> not actual memory usage. In particularly limiting RLIMIT_AS like this
> is actively wrong as this just breaks mmap(). I mean, this is the
> beauty of modern memory management: you can set up mappings, and they
> are relatively cheap and only are backed by real RAM as soon as you
> access them. really, you should not limit RLIMIT_AS, it's not for what
> you think it is.

When would you use this functionality? I struggle to see a use case.

>> That coupled with bug reports often like this one where a PDF with no text
>> is taking over 200 seconds to extract and using 2Gb of memory:
>> https://bugs.freedesktop.org/show_bug.cgi?id=85196
>
> 2Gb of address space or actually dirty memory?

I don't know without looking at poppler sources or further 
investigation. The maintainer says "peak memory usage of 2.7GB." which 
is too ambiguous.

>> That leads to SIGABRT and a bug report against Tracker, which we were not
>> handling because really, the file itself (or library used to open it) should
>> be fixed and tracker-extract was restarting itself until recently (which is
>> a bug that needs fixing).
>>
>> I guess we could use RLIMIT_CPU and handle SIGXCPU, but I have no idea what
>> limitation (in seconds) to use in the first place.
>
> Well, I am pretty sure that if an extractor you fork off takes more
> than 10s you should kill it and skip the file. That's a pretty safe bet.

OK, so you agree with our approach then at least. That's good to know.

Thanks Lennart.

-- 
Regards,
Martyn

Founder & Director @ Lanedo GmbH.
http://www.linkedin.com/in/martynrussell