[systemd-devel] How to use cgroups for Tracker?

[Lennart Poettering]

On Tue, 21.10.14 09:53, Martyn Russell (martyn at lanedo.com) wrote:

> On 20/10/14 21:12, Lennart Poettering wrote:
> >On Tue, 14.10.14 15:35, Martyn Russell (martyn at lanedo.com) wrote:
> 
> Hej Lennart,
> 
> >I am not entirely sure what cgroups would really give you that
> >sched_setscheduler(), ioprio_set(), setrlimit() wouldn't give you
> 
> It's another approach, more of a sandbox in my mind than a priority system
> (which the above APIs are really - possibly with the exception of
> setrlimit()). I will get on to APIs in a bit...
> 
> Tracker has been used on embedded platforms in the past with its own cgroups
> because it was more effective than the other APIs. It was recently suggested
> again in the bug above.

What functionality of cgroups were you precisely using?

> >Anyway, what precisely are you trying to do?
> 
> Even using the kernel APIs, we still get bug reports about crazy CPU use
> and/or disk use. Since employing sched_setscheduler(), I think the
> situation

What precisely are you setting with sched_setscheulder() and ioprio_set()?

> is much better, but some people really want Tracker to
>   a) only perform when they're away from their computer OR
>   b) be completely unnoticeable.
> 
> Now, we can do a) without cgroups, but I believe b) would be better done
> using cgroups.

Well, it will always be noticable since it generates IO. It will
always show up in top/iotop hence. If you want it to not interfere
with other things beind done on the system use
ioprio_set/sched_setscheduler to mark things as batch jobs.

If you want to put hard bandwitdh limits on things, then I#d really
advise not to, because you effectively just prolong with that what you
want to do, keep the hw full ypowered on for longer that way. Hard
bandwidth limits are something for accounting models, for implementing
business plans, but I think it would be wrong to use them for "hiding"
things in top/iotop.

> >Why don't the kernel API calls pointed out above not suffice?
> 
> I think it depends on the API and who you talk to.
> 
> For me, the APIs work quite well. However, we still get bug reports. I find
> this quite hard to quantify personally because the filesystems, hardware and
> version of Tracker all come into play and can make quite some
> difference.

What precisely are the bug reports about?

> The one API that doesn't really work for us is setrlimit(), mainly because
> we have to guess the memory threshold (which we never get right) and we get
> a lot of SIGABRTs that get reported as bugs. I suppose we could catch
> SIGABRT and exit gracefully, but lately, we've agreed (as a team) that if an
> extractor or library we depend on uses 2Gb of memory and brings a smaller
> system to its knees, it's a bug and we should just fix the
> extractor/library, not try to compensate for it. Sadly, there are always
> these sort of bugs and it's precisely why tracker-extract is a separate
> process.

What functionality of setrlimit() are you precisely using?

Lennart

-- 
Lennart Poettering, Red Hat