[systemd-devel] python - reading the journal

[David Strauss]

I don't want this work to get buried. Where are we on the revisions
suggested by Zbyszek?

On Sat, Feb 23, 2013 at 7:34 AM, Zbigniew Jędrzejewski-Szmek
<zbyszek at in.waw.pl> wrote:
> On Sat, Feb 23, 2013 at 02:05:27PM +0000, Steven Hiscocks wrote:
>> On 23/02/13 00:43, Zbigniew Jędrzejewski-Szmek wrote:
>> >On Thu, Feb 21, 2013 at 06:46:35PM +0000, Steven Hiscocks wrote:
>> >>On 21/02/13 01:49, Zbigniew Jędrzejewski-Szmek wrote:
>> >>>On Tue, Feb 19, 2013 at 09:02:56PM +0000, Steven Hiscocks wrote:
>> >>>>I've pushed a few more commits.
>> >>>I've pushed a few commits to
>> >>>https://github.com/keszybz/systemd/commits/python-systemd-reader
>> >>>This is your tree but rebased onto my id128 stuff, and modified
>> >>>to use it. It also has sphinx-generated documentation and gcc
>> >>>all warnings fixed. Please have a look.
>> >>>
>> >>>Zbyszek
>> >>>
>> >>
>> >>Thanks Zbyszek. Looks really good. Tidied it up a bit :)
>> >>
>> >>With the macros for python 2 and 3 differences:
>> >>I wonder if unicode_FromString is a bit confusing, as the macro
>> >>returns a string and not Unicode for python2? Same for long_AsLong,
>> >>which returns int in python2?
>> >I don't think that this matters too much. There's a comment about
>> >the naming convention before the macros, and anyway this is purely
>> >internal stuff.
>> >
>> >>Also, not related to the commits:
>> >>I've noted that `sd_journal_get_cursor` man pages states that it
>> >>returns 0 on success, but it actually returns 1. Note sure if the
>> >>code or man page is in error. There is a comment in _reader.c
>> >>get_next against `sd_journal_get_cursor`, which I guess needs to be
>> >>removed once this is clarified.
>> >This has been now fixed in the master branch.
>> Good stuff :)
>> >
>> >I've pushed a few more commits to
>> >https://github.com/keszybz/systemd/commits/python-systemd-reader.
>> >Important changes:
>> >
>> >1. Checks for return values of all function calls, so now
>> >everything should fail nicely, even on OOM conditions. This
>> >lengthens the code quite a bit, but seems unavoidable.
>> >
>> >2. The value returned for __TIMESTAMP_MONOTONIC:
>> >both fields are returned as a tuple. Under Python 3
>> >a named tuple is used:
>> >
>> >>>>from systemd import journal
>> >>>>t = journal.Journal().get_next()
>> >>>>t['__MONOTONIC_TIMESTAMP'].timestamp
>> >datetime.timedelta(0, 15, 368483)
>> >>>>t['__MONOTONIC_TIMESTAMP'].bootid
>> >UUID('9838676e-2666-4a83-b8be-b0654747f8be')
>> >
>> >I think that this is relatively lightweight, yet nice to use.
>> >
>> >Under Python 2.7, a normal tuple is returned, so it is necessary
>> >to use t['__MONOTONIC_TIMESTAMP'][0], t['__MONOTONIC_TIMESTAMP'][1].
>> >
>> >I first tried with a class, and it felt too complicated.
>> >
>> Like it. Makes sense as the monotonic timestamp is useless without a bootid.
> Right. And we need to export everything that the journal provides.
>
>> I was thinking another way which would allow use of named tuple in
>> both python2.6+ and python3, would be combining the
>> "__MONOTONIC_TIMESTAMP" and "_BOOT_ID" together within the
>> `_convert_entry` method within "journal.py". I assume that the boot
>> id returned by "sd_journal_get_monotonic_usec" will always be the
>> same as "_BOOT_ID"?
> Actually both the time and _BOOT_ID are different, since they
> are both converted (int + bytes to datetime.timedelta + uuid.UUID).
>
>> Or equally you could pass back a plain tuple for
>> __MONOTONIC_TIMESTAMP and then change namedtuple within python
>> _convert_entry?
> Hm, that would be possible. Or maybe just use the named tuple
> just when the C PyStructSequnce is not available. I think that
> (like in most of systemd) we should concentrate on the latest
> version (Python 3.3 and 3.4), with fallbacks for legacy which
> don't have to be as nice.
>
>> How about changing the `seek_monotonic` method in journal.py to
>> accept a tuple?
> I think that doesn't gain anything, but makes manual calls
> minimally harder. With a tuple one can always say
>  .seek_monotonic(*timestamp_with_boot_id)
> which is nice and explicit.
>
>> >3. Sphinx documentation can be generated with 'make destdir-sphinx'.
>> >This target is part of 'make doc-sync', which means that the documents
>> >will be uploaded along with the man pages to freedesktop.org.
>> >So when this is merged
>> >   http://www.freedesktop.org/software/systemd/man/python-systemd
>> >will be the URL with docs. Preview is at:
>> >   http://keszybz.github.com/systemd/python-systemd/
>> >
>> >This way that the docs are generated is not very nice (whole package
>> >is installed using random DESTDIR, and sphinx-build is run with
>> >PYTHONPATH and LD_LIBRARY_PATH pointing to this directory), but seems
>> >to work. The documentation is only generated on request, i.e. only by
>> >conscious developers, so it's not catastrophic even if this doesn't
>> >always work.
>> >
>> >I think that this is becoming mergeable.
>> >
>> >Zbyszek
>> >
>> Thanks for all your work on this Zbyszek :)
> :)
>
> Zbyszek



--
David Strauss
   | david at davidstrauss.net
   | +1 512 577 5827 [mobile]