[systemd-devel] [Tracker] How to use cgroups for Tracker?

[Philip Van Hoof]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 23/10/2014 8:15, Aleksander Morgado wrote:
> On Thu, Oct 23, 2014 at 1:24 AM, Philip Van Hoof
> <philip at codeminded.be> wrote:

[cut]

>>> This is fixable, by enforcing a size limit on the queue. As
>>> the limit is hit the algorithm should coalesce queued events
>>> based on subtrees. For example, if one event for /foo/bar/buzz
>>> and one for /foo/bar/waldo is queued, and the queue hits its
>>> limit, both are replaced by an entry for /foo/bar which is
>>> marked with a new flag that some event was lost below this
>>> subtree. For clients this would then mean that when they
>>> receive this they must rescan that specific subtree again, but
>>> not the whole tree.
>>> 
>>> It's a simple algorithm, the max number of entries stays
>>> fixed, but perfomance doesn't go completely horrible when the
>>> limit is reached.
>>> 
>>> Such a scheme should be implemented in fanotify on the kernel 
>>> side.


[cut]

>> But yes. The coalesce solution in fanotify would be a good idea
>> to allow unprivileged processes. Probably better than what
>> FSEvents does.
> 
> That's a good one indeed; coalescing events in that way in the
> kernel looks quite a sane approach. Still, one single process in
> userspace doing all the control of what changed when (like FSEvents
> does) may actually behave much nicer, as other processes could ask
> for all changes coalesced since a specific timestamp (e.g. since
> that process was run). Not thinking in Tracker here, think of a
> program which runs sporadically, but when it runs it wants to know
> what changed since the last time it was run (e.g. a backup app).

I'd design that API to be a lot like how tracker:modified and several
protocol's modseqs works: store a modification sequence in each new
record of the journal and return it after each query by a client.

Clients can then using the modification sequence ask for all changes
since that number.

However. This also means logging forever. So the same problems
tracker-store has when you don't disable its journal: ie. always
growing storage for it and privacy - the 'if I remove/rename/create a
file, I don't want its existence to be nevertheless logged
forever'-use-case. For example if the administrator takes away +rx
from a directory for a user, the user could with the journal still
know what used to be there. So we'd need a method for the admin to
purge that.

I think it's inevitable that such log files will need to be truncated,
rotated and/or also size limited at some point.

The coalesce-only solution doesn't have this problem.

> Anyway, please remember that being privileged isn't the only
> reason why Tracker can't use fanotify. It's API being fd-based, it
> works on existing open files only; e.g. it won't notify file
> deletes or move events, among other things. If we want some
> recursirve monitoring approach with all CREATE/UPDATE/DELETE/MOVE
> events, something new needs to be implemented, or inotify somehow
> improved to handle that.


Correct.

Kind regards,

Philip



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (MingW32)

iQEcBAEBAgAGBQJUSLIdAAoJEEP2NSGEz4aDrEgIAML+jfWgzRrnnTop7jRrLXdv
bRjibU37FaK45rauUxX395ye13hqUWQYUx9FVE4xh6y/yvJVFGCt1SfJFuGIu8+S
XLaud2qY145ePIEfakD4z3iX9BsAWfsWIcvGu6QIz7tpT8Nd6SM+tJYwRx9V1dli
vdspQh8Kf1Xg9DxUjmx2bnVYzV7KFM3dRgyuOGrxlHVjAGOH9HFAn1n0PJlO87Fu
8Hzr2vLgRU+zDnFY8jifavLKo4+a6/2UHeLbthhwRUUNBk9LZixblCs5wgAPJmb3
SZdIk5u+EqOYCAwWs6AgIEl/XQMbiC34DVS+rtckm0DNA8lmvzOTvvPar/ZkuzE=
=Ldnv
-----END PGP SIGNATURE-----