[Telepathy] thinking about a new log format for telepathy-logger

[Ivan Vučica]

Why not SQLite, as proposed in 4)? Log data can always be easily  
dumped into XML if deemed necessary, but this way it's perhaps more  
easily searchable and filtered? E.g. "grab all data generated by pecan  
between 1am and 2am today". "Return per-hour message count stats for  
gabble and pecan".

Regards,
Ivan Vucica
via phone

On 15. ožu. 2010., at 02:10, Danielle Madeley <danielle.madeley at collabora.co.u 
k> wrote:

> So Telepathy Logger currently logs things in something not unlike
> Empathy's XML log format. Although XML has some advantages (like being
> able to generate logs using XSL), it seems fairly sub-optimal for the
> efficient storage of logs.
>
> I've been thinking (and playing) with some ideas for how to replace
> this, and am looking for feedback/ideas.
>
> Solution #1: serialising binary structs
>
>        This is where we serialise a fixed struct directly into the
>        file. Something like:
>
>        [danni at adelie log-backend]$ xxd test
>        0000000: 0000 0060 0000 0000 4b9d 7725 0000 0003  ...`....K.w
>        %....
>        0000010: 0000 0020 6461 6e69 656c 6c65 2e6d 6164  ...
>        danielle.mad
>        0000020: 656c 6579 4063 6f6c 6c61 626f 7261 2e63
>        eley at collabora.c
>
>        This message is formatted (guint32 entry-length)(gint64
>        timestamp)(guint32 flags)((guint32 string-length)(string id)).
>
>        Not really a fan of doing this, because it makes it quite
>        difficult to extend later on (especially to non-text messages).
>
> Solution #2: binary tag-based thing (similar to Apple DMAP)
>
>        This is also a binary format, but a tag based one where there  
> is
>        a table of tags, where each tag has a length and a data type
>        (that type can be Container). Unpacked it might look something
>        like this:
>
>        (Format TAG length ..data..)
>
>        LOGM 139
>          NAME 16 "Danielle Madeley"
>          IDXX 32 "danielle.madeley at collabora.co.uk"
>          TIME 8  1268610853
>          FLAG 4  0x3
>          MESG 39 "This is a message that is 39 bytes long"
>
>        Types:
>
>        LOGM container "Log Message"
>        NAME string    "Name"
>        IDXX string    "Id"
>        TIME gint64    "Timestamp"
>        FLAG guint32   "Flags"
>        MESG string    "Message"
>
>        Packs something like this (exciting intermix of ascii and hex):
>
>        LO GM 008B NA ME 0010 Da ni el le _M ad el ey ID XX 0020 da ni
>        el le .m ad el ey @c ol la bo ra .c o. uk TI ME 0008 0000 0000
>        4B9D 7725
>
>        I kind of like this format because it's compact and extensible.
>        It's quick to jump from container to container. However it's
>        still very custom.
>
> Solution #3: EXI or similar
>
>        Basically use some binary XML format. This is more or less a
>        formalisation of the system proposed above. The question is
>        which format? They all seem incredibly overengineered for our
>        purposes.
>
> Solution #4: sqlite
>
>        Store each message as an SQLite row. Great for searching,
>        probably won't scale?
>
> Thoughts?
>
> --danielle
>
> -- 
> Danielle Madeley
> Software Developer, Collabora Ltd.                  Melbourne,  
> Australia
>
> www.collabora.co.uk
>
> _______________________________________________
> telepathy mailing list
> telepathy at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/telepathy