[systemd-devel] Client logging to journald without libsystemd-journal.so

[Daniel P. Berrange]

I recently introduced support for libvirt logging to journald. Initially I
had intended to use libsystemd-journal.so for the logging, however, in the
end I made libvirt directly communicate with sendmsg().

First, I wanted to confirm two interface stability issues.

 - Is the client app -> journald logging protocol considered to be
   ABI stable ?
 - Is the /run/systemd/journal/socket path considered to be stable ?

Second, I wanted to mention why we couldn't use libsystemd-journal.so
ourselves.

The first problem is that there is no sd_journal_open/close API call
to setup the file descriptor. The library uses a one time atomic
global initialize to open its file descriptor which is then cached
until exit() or execve() (it has SOCK_CLOEXEC set).

The problem is that when libvirt does fork() to create client processes,
one of the things it does is to iterate from 0 -> sysconf(_SC_OPEN_MAX),
closing every file descriptor, except those in its whitelist.

Now I know there is the school of thought that says this is a bad idea,
and that all code should correctly set O_CLOEXEC for all file descriptors.
While a nice idea in theory, unfortunately this is not really practical
for us in reality because there are too many 3rd party libraries we use
which don't do this correctly. Not least because traditional UNIX APIs
don't allow for atomically creating an FD with O_CLOEXEC set. So we're
stuck with closing all FDs after fork() for a good long time yet.

There are two things libsystemd-journal could do to help apps in this
scenario. Either provide a way for apps to query the cached journal
logging file descriptor, allowing them to explicitly leave it open.
Alternatively provide explicit API to call to re-open the FD, which
they could call after fork(). Possibly other solutions too, like
requiring an explicit close/open like syslog though that has its own
set of problems.

The second blocker problem was figuring out a way to send log messages
using only APIs declared async-signal safe. Again this is so that we
can safely send log messages inbetween fork() and execve() which only
permits async signal safe APIs. The sd_journal_send() API can't be
used since it relies on vasprintf() which can allocate using malloc.

The sd_journal_sendv() API is pretty close to what we'd want, but
the way you have to format the iovec doesn't quite work. IIUC, it
requires that each iovec contains a single formatted log item
string "KEY=VALUE". Populating data in such a way is inconvenient
for libvirt. For libvirt it was easier for us to use two iovec
elements for each log item, "KEY=" and "VALUE", so that we can
avoid doing the data copy implied by filling a single string with
"KEY=VALUE".

The upshot is that we ended up filling an iovec[] ourselves, taking
care of escaping '\n', and then directly sending it to journald.

As long as the wire format and UNIX socket path are considered ABI
stable by systemd devs, I'm fairly happy with the libvirt code as
it. I just mention these issues in case you think it is desirable
to add further libsystemd-journal.so APIs to make life easier for
other applications doing logging in the future.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|