[systemd-devel] Journald logging handler for Python 3 and AsyncIO integration

Ludovic Gasc gmludo at gmail.com
Mon May 4 14:33:54 PDT 2015


2015-05-04 3:16 GMT+02:00 Lennart Poettering <lennart at poettering.net>:

> On Sat, 02.05.15 15:12, Ludovic Gasc (gmludo at gmail.com) wrote:
>
> >    2. We use heavily AsyncIO module to have async pattern in Python,
> >    especially for I/O: https://docs.python.org/3/library/asyncio.html
> >    In the source code of python-systemd, I've seen that you use a C glue
> to
> >    interact with journald, but I don't understand what's the
> communication
> >    between my Python daemon process and journald: unix sockets ? Other
> >    mechanism ? Depends on the mechanism, it should be have an impact for
> us.
>
> The communication is via an AF_UNIX/SOCK_DGRAM socket in the file
> system.
>
> I am very sure that logging should not be asynchronous non-blocking
> IO. It's about reliably getting out log messages at the right times,
> and that's a property you lose if you enqueue logs non-blocking.
>
> I don't think that's a good idea in any programming language to log
> asynchronously. I mean, there's a reason why libc syslog() is
> blocking too.
>

Yes, you're right.
The only way I see to send asynchronously messages and keep the order is to
add a precise timestamp in the message.
But the "time" you need to access to the clock and push that on the wire,
I'm not sure you'll be more efficient, without to count the cost of the
effort to implement that in journald.


> If you are afraid of blocking logging, then make sure to use large
> socket buffers. the journal client library and journald will try to
> use very large buffers, but this doesn't always work if the client is
> unprivileged. Also you might have to increase the number of datagrams
> that may be queued with /proc/sys/net/unix/max_dgram_qlen
>

Thanks for the tip, I didn't know that.
Ok, it seems to be a more pragmatic and efficient way, I'll do that and
validate with a benchmark to be sure.

BTW, FYI, I don't find a solution to install python-systemd in a pyvenv.
Finally, I've "reverse-engineered" journald "protocol" with socat.
The protocol is so simple that I've written a tiny library written in pure
Python to be easy to install.

I'll opensource that soon.

Thanks for your help.

Regards.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/systemd-devel/attachments/20150504/cdd6b432/attachment.html>


More information about the systemd-devel mailing list