[systemd-devel] [Tracker] How to use cgroups for Tracker?

[Philip Van Hoof]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 23/10/2014 1:03, Lennart Poettering wrote:
> On Thu, 23.10.14 00:52, Philip Van Hoof (philip at codeminded.be)
> wrote:
> 
>>> Don't try to work around limitations of kernel APIs by 
>>> implementing inherently not scalabale algorithms in userspace.
>>> I mean, you implemented something that scales O(n) with n the
>>> numbers of dirs. That's what you need to fix, there's no way
>>> around that. Just looking for magic wands in cgroups and
>>> scheduling facilities to make an algorithm that fundamentally
>>> scales badly acceptable is not going to work.
>> 
>> The problem with allowing unprivileged processes is that a
>> misbehaving process will cause the kernel to buffer events for
>> it, forcing the kernel to dynamically allocate memory. Not
>> something kernel or inotify developers will like a lot.
> 
> This is fixable, by enforcing a size limit on the queue. As the
> limit is hit the algorithm should coalesce queued events based on 
> subtrees. For example, if one event for /foo/bar/buzz and one for 
> /foo/bar/waldo is queued, and the queue hits its limit, both are 
> replaced by an entry for /foo/bar which is marked with a new flag
> that some event was lost below this subtree. For clients this would
> then mean that when they receive this they must rescan that
> specific subtree again, but not the whole tree.
> 
> It's a simple algorithm, the max number of entries stays fixed,
> but perfomance doesn't go completely horrible when the limit is
> reached.
> 
> Such a scheme should be implemented in fanotify on the kernel
> side.

And clients must be prepared to scan coalesced queued events, which
can be large (when /foo/bar/waldo and /foo/bar/buzz are directories
with large amounts of files, coalescing to /foo/bar means
tracker-miner-fs suddenly has a lot of work to do) ..

But yes. The coalesce solution in fanotify would be a good idea to
allow unprivileged processes. Probably better than what FSEvents does.

Kind regards,

Philip

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (MingW32)

iQEcBAEBAgAGBQJUSDzAAAoJEEP2NSGEz4aD6I8H/i/POL3hIfCCz/wwA+F6Kwol
ytuFKWXYedtfApkzbbeX8aH+CYiTZfq9Ld46MVgKzz3oydD5JSsnFVns33yuumW8
AZ66yhZWTJZnfCDE8ZJnLq5/lxyfgZq0AOlfLW9loXwQ54zPLBqFztSdtDr7EXMS
2dQ134h9N2em4vdYV0p3hwCKSwifOdCZRi2xMvr03RGPDTTsnvMN9Ou6gCEscTEA
fjNATi969kfjQsf08C5VVVwemyU8pf5RU725ZfoQmpMamJcTREVSrjt/qpno7F03
5rAoJWXCkTEGl56cbeQJX0n54m45Ce6XHpPJLfkouKoxGWhNVg8ThU/Pw0GKHZs=
=slie
-----END PGP SIGNATURE-----