[systemd-devel] Antw: [EXT] Journalctl exits under heavy logs

[Ulrich Windl]

>>> Ravindran Shanmugam <ravindran.shan at gmail.com> schrieb am 20.03.2021 um
20:50
in Nachricht
<CAC4QDGdbDvhnJ3E+mCDr+s3GZQ+eztLbD=fQUKGOUeeQy=Y_ow at mail.gmail.com>:
> Hi,
> 
> 
> 
> Bug / Issue:
> 
> Under heavy logs entering the systemd-journald, journalctl Exits with one
> of the following error messages:
> 
> 
> 
> Failed to iterate through journal: Bad message
> 
> Failed to get realtime timestamp: Bad message
> 
> Failed to get monotonic timestamp: Bad message

In theory with recent machines, you can get two identical timestamps even with
nanosecond resolution on subsequent calls.
I learned that some months ago.

> 
> 
> 
> The systemd is at version 243
> 
> 
> 
> Is this an Upstream bug which was fixed.?
> 
> 
> 
> N.B:-
> 
> As of now, not planning to upgrade to latest systemd version(247) for some
> legacy reasons.
> 
> 
> 
> 
> 
> Repro steps:-
> 
> 1)Change the following 2 options in the file "/etc/systemd/journald.conf"
> from
> 
> 
> 
> RateLimitIntervalSec=1s
> 
> RateLimitBurst=2000
> 
>   to
> 
> RateLimitIntervalSec=0
> 
> RateLimitBurst=0
> 
> 
> 
> to turn off rate limiting, and then kill the "systemd-journald" process so
> that the new one
> 
> will be spawned and picks up the new jounald.conf
> 
> 
> 
> 
> 
> 
> 
> 2) Start ‘journalctl’ with this command:
> 
>   "/bin/journalctl -o json
>
--output-fields=_SOURCE_REALTIME_TIMESTAMP,__REALTIME_TIMESTAMP,SYSLOG_IDENT
> IFIER,_SYSTEMD_UNIT,SYSLOG_FACILITY,MESSAGE,PRIORITY
> -f --no-tail > /dev/null 2>&1 &"
> 
> 
> 
> 
> 
> 
> 
> 3)Create following bash script and run it as "test.sh 100 1"
> 
> --------------------
> 
> #! /bin/bash
> 
> 
> 
> if [[ ! -e /tmp/lines1.txt ]]; then
> 
>     for i in $(seq 1 50001); do echo "abc1-$i" >> /tmp/lines1.txt; done
> 
>     for i in $(seq 1 50001); do echo "abc2-$i" >> /tmp/lines2.txt; done
> 
>     for i in $(seq 1 50001); do echo "abc3-$i" >> /tmp/lines3.txt; done
> 
>     for i in $(seq 1 50001); do echo "abc4-$i" >> /tmp/lines4.txt; done
> 
>     for i in $(seq 1 50001); do echo "abc5-$i" >> /tmp/lines5.txt; done
> 
> fi
> 
> 
> 
> loggernum=${1}
> 
> if [[ -z ${loggernum} ]]; then
> 
>     loggernum=10
> 
> fi
> 
> loopnum=$((loggernum/5+1))
> 
> 
> 
> foreverloop='no'
> 
> if [[ -n "$2" ]]; then
> 
>     foreverloop='yes'
> 
> fi
> 
> 
> 
> while [[ 1 ]]; do
> 
>     for i in $(seq 1 ${loopnum}); do
> 
>         logger -p local0.3 -f /tmp/lines1.txt &
> 
>         logger -p local0.3 -f /tmp/lines2.txt &
> 
>         logger -p local0.3 -f /tmp/lines3.txt &
> 
>         logger -p local0.3 -f /tmp/lines4.txt &
> 
>         logger -p local0.3 -f /tmp/lines5.txt &
> 
>     done
> 
> 
> 
>     sleep 1
> 
>     ps aux | grep logger | wc -l
> 
> 
> 
>     if [[ "${foreverloop}" == "yes" ]]; then
> 
>         echo 'wait for loggers to exit'
> 
>         wait
> 
>         journalctl_pid=$(pidof journalctl)
> 
>         if [[ -z "${journalctl_pid}" ]]; then
> 
>             echo "journalctl dead, exit"
> 
>             exit 1
> 
>         fi
> 
>     else
> 
>         exit 0
> 
>     fi
> 
> done
> 
> 
> 
> 
> 
> 4) Wait for until the message "journalctl dead, exit" appears on the
> console, this means journalclt exited.
> 
> 
> 
> 
> 
> Rgds,
> 
> --Ravi