[Bug 49687] New event based LogStore

[bugzilla-daemon at freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=49687

--- Comment #6 from Seif Lotfy <seif at lotfy.com> 2012-05-09 09:55:41 PDT ---
(In reply to comment #5)
> (In reply to comment #3)
> > Good point. How would you propose solving the issue though. I mean with or
> > without Zeitgeist if the logger exists we lose the message. Or did I miss
> > something?
> 
> Seif, the problem is due to fact that the logged info is split in two: Event
> Log and Body Index.
> Without ZG (the Event Log), there would be just one sync call to Sqlite, to
> store the message at once.
> 
> We are talking of a clean exit, where for example TPL times out after some time
> without channel activities (I don't think it acts that way yet, but it was
> supposed to).

Very good point.

One of the main reasons I thought of splitting the DB is to wrap the TBI
(message DB) is to be able to fullindex the messages, with and FTS library such
as Xapian or Lucene.

Having the Events and the Messages in one DB just makes it not so clean in
terms of separation of concern. What if we decide that Xapian is not so good of
a FTS indexer. This would mean wrapping the whole DB all over again.

Anyhow we could create a dedicated integrated event DB. Which does its own sync
read/write for the Log. We could do that with a our own integration or with the
upcoming libzeitgeist2 which creates a DB for you without the daemon. In both
cases if your worries are doing stuff async then swtiching to sync writing
should not be an issue.

> The bigger issue is that the second part (body) is the one most likely to be
> lost and it's also the most important (or at least equally important with some
> other event's info).

When can it get lost.

> I don't care if I don't remember the avatar used with the message or the
> geolocation of the event.
> I care if I don't have the body or I cannot associate the it with timestamp or
> from/to.
> 
> Is there any way to invert the process?
> 
> 1- write the body with the minimum set of needed info into the Body Index (even
> if duplicated in the Log later), assigning a primary key X
> 2- write the Log, telling ZG that this event is related to X (or giving it our
> own event_id).

Sadly you can't give Zeitgeist an event_id to an event.

Well a good solution is to have a temp_table which stores all the info as it is
(strings) as soon as they arrive. When an interaction happens we will first
dump it in the temp_table. Then we insert into the log then into the TBI. Once
both insertions took place we remove from the temp_table. This way if TPL quits
or crashes, or the Log is not reachable the middle of a process, the next time
tpl start it will find the temp_table not-empty and try to empty it. You might
ask why not keep the temp_table as our main storage. Well:
1) it is hard to do a FTS index around it.
2) it will have duplicate string entries for example the target string. Which
can be costly and should be rather stored as an int.

> This is a scenario in which we have a private DB for what we need and delegate
> to ZG all the extra data, rather to have the private DB to keep what ZG cannot
> store/is better not store in ZG.

I can't follow. Can you elaborate?

> Seif, on a (not so) unrelated subject: can you explain how you thought to
> manage the "delete event" API? It might be insightful for this problem.
> Scenario: TPL's event_id_X is deleted from ZG while TPL is not running?
> 
> I don't remember it exactly, but it was a sort of recovery procedure. Can it be
> applied in this case? (this would cover the situation for which TPL exists
> before ZG returns, not in case ZG does not answer though).

Well in case we will have a Zeitgeist extension that does nothing else but
listen to deleted events and stores the deleted event_ids. Once TPL runs again
it requests all the deleted events from Zeitgeist, and update its TBI.

-- 
Configure bugmail: https://bugs.freedesktop.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA Contact for the bug.
You are the assignee for the bug.