[systemd-devel] journal and /run/systemd/journal/syslog

[Umut Tezduyar]

On Tue, Jul 9, 2013 at 1:58 AM, Lennart Poettering
<lennart at poettering.net> wrote:
> On Sun, 07.07.13 19:18, Umut Tezduyar (umut at tezduyar.com) wrote:
>
>> Hi,
>>
>> I have noticed that if journal fails to send messages to
>> /run/systemd/journal/syslog, it increments a counter and every 30
>> seconds it prints out a message on journal informing that forwarding x
>> many messages has failed. If journal storage is set to none, there is
>> no way of being informed about the missing messages. Should we not try
>> to send a message to /run/systemd/journal/syslog if/when it has space
>> again?
>
> Yes, sounds sensible. Added to the TODO list.
>
>> syslog.socket sets ReceiveBuffer=8M. My understanding is intention of
>> setting this value is specifying the maximum size of one datagram
>> message that is allowed to send. socket.c is setting the SO_RCVBUF on
>> the socket. According to man unix 7, SO_RCVBUF has no effect on the
>> datagram but SO_SNDBUF does. Am I missing something?
>
> Oh, hm, this sucks. I wasn't aware of this. I always assumed SO_RCVBUF
> and SO_SNDBUF would both alter the same buffer, just from differente
> "sides".
>
>> Number of messages that can be queued on /run/systemd/journal/syslog
>> is controlled by /proc/sys/net/unix/max_dgram_qlen and this value is
>> by default 10 in most distributions. What this means is, after journal
>> forwards 11 messages to the socket, it will not be able to do so until
>> external logging daemon comes up and clears up
>> /run/systemd/journal/syslog. If logging daemon is starting after
>> basic.target, it is pretty certain that it will not be able to receive
>> all the early boot messages. Increasing
>> /proc/sys/net/unix/max_dgram_qlen is an option but is increasing a
>> system wide limit right thing to do?
>
> Yeah, this is a big issue. For a long time there has been an item on the
> kernel wishlist to get an ioctl for setting the qlen per-socket. The
> code for that should actually be simple.

Luckily kernel doesn't allocate a buffer for the socket
(kernel:net/unix/af_unix.c) according to the value meaning setting it
to 10 or 1000 doesn't matter in terms of pre-allocating memory.

>
> We though about just bumping this globally via sysctl, but we feared
> that might not sit well with some folks, as we shouldn't change a global
> setting just because one user of it would benefit, especially given that
> we don't know the effect this might have on others...

I want to go this route and I think it is not possible at the moment
due to undetermined start order between syslog.socket and
systemd-sysctl.service. Can we change that?

>
> The better approach really appeared to us to be fixing the kernel
> properly instead, and get that new ioctl.
>
> Lennart
>
> --
> Lennart Poettering - Red Hat, Inc.