[systemd-devel] Journalctl exits under heavy logs

[Ravindran Shanmugam]

Hi,



Bug / Issue:

Under heavy logs entering the systemd-journald, journalctl Exits with one
of the following error messages:



Failed to iterate through journal: Bad message

Failed to get realtime timestamp: Bad message

Failed to get monotonic timestamp: Bad message



The systemd is at version 243



Is this an Upstream bug which was fixed.?



N.B:-

As of now, not planning to upgrade to latest systemd version(247) for some
legacy reasons.





Repro steps:-

1)Change the following 2 options in the file "/etc/systemd/journald.conf"
from



RateLimitIntervalSec=1s

RateLimitBurst=2000

  to

RateLimitIntervalSec=0

RateLimitBurst=0



to turn off rate limiting, and then kill the "systemd-journald" process so
that the new one

will be spawned and picks up the new jounald.conf







2) Start ‘journalctl’ with this command:

  "/bin/journalctl -o json
--output-fields=_SOURCE_REALTIME_TIMESTAMP,__REALTIME_TIMESTAMP,SYSLOG_IDENTIFIER,_SYSTEMD_UNIT,SYSLOG_FACILITY,MESSAGE,PRIORITY
-f --no-tail > /dev/null 2>&1 &"







3)Create following bash script and run it as "test.sh 100 1"

--------------------

#! /bin/bash



if [[ ! -e /tmp/lines1.txt ]]; then

    for i in $(seq 1 50001); do echo "abc1-$i" >> /tmp/lines1.txt; done

    for i in $(seq 1 50001); do echo "abc2-$i" >> /tmp/lines2.txt; done

    for i in $(seq 1 50001); do echo "abc3-$i" >> /tmp/lines3.txt; done

    for i in $(seq 1 50001); do echo "abc4-$i" >> /tmp/lines4.txt; done

    for i in $(seq 1 50001); do echo "abc5-$i" >> /tmp/lines5.txt; done

fi



loggernum=${1}

if [[ -z ${loggernum} ]]; then

    loggernum=10

fi

loopnum=$((loggernum/5+1))



foreverloop='no'

if [[ -n "$2" ]]; then

    foreverloop='yes'

fi



while [[ 1 ]]; do

    for i in $(seq 1 ${loopnum}); do

        logger -p local0.3 -f /tmp/lines1.txt &

        logger -p local0.3 -f /tmp/lines2.txt &

        logger -p local0.3 -f /tmp/lines3.txt &

        logger -p local0.3 -f /tmp/lines4.txt &

        logger -p local0.3 -f /tmp/lines5.txt &

    done



    sleep 1

    ps aux | grep logger | wc -l



    if [[ "${foreverloop}" == "yes" ]]; then

        echo 'wait for loggers to exit'

        wait

        journalctl_pid=$(pidof journalctl)

        if [[ -z "${journalctl_pid}" ]]; then

            echo "journalctl dead, exit"

            exit 1

        fi

    else

        exit 0

    fi

done





4) Wait for until the message "journalctl dead, exit" appears on the
console, this means journalclt exited.





Rgds,

--Ravi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20210321/dcf7d6b9/attachment.htm>