[systemd-devel] python - reading the journal

[Zbigniew Jędrzejewski-Szmek]

On Sat, Feb 23, 2013 at 02:05:27PM +0000, Steven Hiscocks wrote:
> On 23/02/13 00:43, Zbigniew Jędrzejewski-Szmek wrote:
> >On Thu, Feb 21, 2013 at 06:46:35PM +0000, Steven Hiscocks wrote:
> >>On 21/02/13 01:49, Zbigniew Jędrzejewski-Szmek wrote:
> >>>On Tue, Feb 19, 2013 at 09:02:56PM +0000, Steven Hiscocks wrote:
> >>>>I've pushed a few more commits.
> >>>I've pushed a few commits to
> >>>https://github.com/keszybz/systemd/commits/python-systemd-reader
> >>>This is your tree but rebased onto my id128 stuff, and modified
> >>>to use it. It also has sphinx-generated documentation and gcc
> >>>all warnings fixed. Please have a look.
> >>>
> >>>Zbyszek
> >>>
> >>
> >>Thanks Zbyszek. Looks really good. Tidied it up a bit :)
> >>
> >>With the macros for python 2 and 3 differences:
> >>I wonder if unicode_FromString is a bit confusing, as the macro
> >>returns a string and not Unicode for python2? Same for long_AsLong,
> >>which returns int in python2?
> >I don't think that this matters too much. There's a comment about
> >the naming convention before the macros, and anyway this is purely
> >internal stuff.
> >
> >>Also, not related to the commits:
> >>I've noted that `sd_journal_get_cursor` man pages states that it
> >>returns 0 on success, but it actually returns 1. Note sure if the
> >>code or man page is in error. There is a comment in _reader.c
> >>get_next against `sd_journal_get_cursor`, which I guess needs to be
> >>removed once this is clarified.
> >This has been now fixed in the master branch.
> Good stuff :)
> >
> >I've pushed a few more commits to
> >https://github.com/keszybz/systemd/commits/python-systemd-reader.
> >Important changes:
> >
> >1. Checks for return values of all function calls, so now
> >everything should fail nicely, even on OOM conditions. This
> >lengthens the code quite a bit, but seems unavoidable.
> >
> >2. The value returned for __TIMESTAMP_MONOTONIC:
> >both fields are returned as a tuple. Under Python 3
> >a named tuple is used:
> >
> >>>>from systemd import journal
> >>>>t = journal.Journal().get_next()
> >>>>t['__MONOTONIC_TIMESTAMP'].timestamp
> >datetime.timedelta(0, 15, 368483)
> >>>>t['__MONOTONIC_TIMESTAMP'].bootid
> >UUID('9838676e-2666-4a83-b8be-b0654747f8be')
> >
> >I think that this is relatively lightweight, yet nice to use.
> >
> >Under Python 2.7, a normal tuple is returned, so it is necessary
> >to use t['__MONOTONIC_TIMESTAMP'][0], t['__MONOTONIC_TIMESTAMP'][1].
> >
> >I first tried with a class, and it felt too complicated.
> >
> Like it. Makes sense as the monotonic timestamp is useless without a bootid.
Right. And we need to export everything that the journal provides.

> I was thinking another way which would allow use of named tuple in
> both python2.6+ and python3, would be combining the
> "__MONOTONIC_TIMESTAMP" and "_BOOT_ID" together within the
> `_convert_entry` method within "journal.py". I assume that the boot
> id returned by "sd_journal_get_monotonic_usec" will always be the
> same as "_BOOT_ID"?
Actually both the time and _BOOT_ID are different, since they
are both converted (int + bytes to datetime.timedelta + uuid.UUID).

> Or equally you could pass back a plain tuple for
> __MONOTONIC_TIMESTAMP and then change namedtuple within python
> _convert_entry?
Hm, that would be possible. Or maybe just use the named tuple
just when the C PyStructSequnce is not available. I think that
(like in most of systemd) we should concentrate on the latest
version (Python 3.3 and 3.4), with fallbacks for legacy which
don't have to be as nice.
 
> How about changing the `seek_monotonic` method in journal.py to
> accept a tuple?
I think that doesn't gain anything, but makes manual calls
minimally harder. With a tuple one can always say
 .seek_monotonic(*timestamp_with_boot_id)
which is nice and explicit.

> >3. Sphinx documentation can be generated with 'make destdir-sphinx'.
> >This target is part of 'make doc-sync', which means that the documents
> >will be uploaded along with the man pages to freedesktop.org.
> >So when this is merged
> >   http://www.freedesktop.org/software/systemd/man/python-systemd
> >will be the URL with docs. Preview is at:
> >   http://keszybz.github.com/systemd/python-systemd/
> >
> >This way that the docs are generated is not very nice (whole package
> >is installed using random DESTDIR, and sphinx-build is run with
> >PYTHONPATH and LD_LIBRARY_PATH pointing to this directory), but seems
> >to work. The documentation is only generated on request, i.e. only by
> >conscious developers, so it's not catastrophic even if this doesn't
> >always work.
> >
> >I think that this is becoming mergeable.
> >
> >Zbyszek
> >
> Thanks for all your work on this Zbyszek :)
:)

Zbyszek