<div dir="ltr">Thanks, but how does that relate to these failed messages?</div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 22, 2021 at 1:47 PM Ulrich Windl <<a href="mailto:Ulrich.Windl@rz.uni-regensburg.de">Ulrich.Windl@rz.uni-regensburg.de</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">>>> Ravindran Shanmugam <<a href="mailto:ravindran.shan@gmail.com" target="_blank">ravindran.shan@gmail.com</a>> schrieb am 20.03.2021 um 20:50 in Nachricht <CAC4QDGdbDvhnJ3E+mCDr+s3GZQ+eztLbD=fQUKGOUeeQy=<a href="mailto:Y_ow@mail.gmail.com" target="_blank">Y_ow@mail.gmail.com</a>>: > Hi, > > > > Bug / Issue: > > Under heavy logs entering the systemd-journald, journalctl Exits with one > of the following error messages: > > > > Failed to iterate through journal: Bad message > > Failed to get realtime timestamp: Bad message > > Failed to get monotonic timestamp: Bad message In theory with recent machines, you can get two identical timestamps even with nanosecond resolution on subsequent calls. I learned that some months ago. > > > > The systemd is at version 243 > > > > Is this an Upstream bug which was fixed.? > > > > N.B:- > > As of now, not planning to upgrade to latest systemd version(247) for some > legacy reasons. > > > > > > Repro steps:- > > 1)Change the following 2 options in the file "/etc/systemd/journald.conf" > from > > > > RateLimitIntervalSec=1s > > RateLimitBurst=2000 > > to > > RateLimitIntervalSec=0 > > RateLimitBurst=0 > > > > to turn off rate limiting, and then kill the "systemd-journald" process so > that the new one > > will be spawned and picks up the new jounald.conf > > > > > > > > 2) Start ‘journalctl’ with this command: > > "/bin/journalctl -o json > --output-fields=_SOURCE_REALTIME_TIMESTAMP,__REALTIME_TIMESTAMP,SYSLOG_IDENT > IFIER,_SYSTEMD_UNIT,SYSLOG_FACILITY,MESSAGE,PRIORITY > -f --no-tail > /dev/null 2>&1 &" > > > > > > > > 3)Create following bash script and run it as "test.sh 100 1" > > -------------------- > > #! /bin/bash > > > > if [[ ! -e /tmp/lines1.txt ]]; then > > for i in $(seq 1 50001); do echo "abc1-$i" >> /tmp/lines1.txt; done > > for i in $(seq 1 50001); do echo "abc2-$i" >> /tmp/lines2.txt; done > > for i in $(seq 1 50001); do echo "abc3-$i" >> /tmp/lines3.txt; done > > for i in $(seq 1 50001); do echo "abc4-$i" >> /tmp/lines4.txt; done > > for i in $(seq 1 50001); do echo "abc5-$i" >> /tmp/lines5.txt; done > > fi > > > > loggernum=${1} > > if [[ -z ${loggernum} ]]; then > > loggernum=10 > > fi > > loopnum=$((loggernum/5+1)) > > > > foreverloop='no' > > if [[ -n "$2" ]]; then > > foreverloop='yes' > > fi > > > > while [[ 1 ]]; do > > for i in $(seq 1 ${loopnum}); do > > logger -p local0.3 -f /tmp/lines1.txt & > > logger -p local0.3 -f /tmp/lines2.txt & > > logger -p local0.3 -f /tmp/lines3.txt & > > logger -p local0.3 -f /tmp/lines4.txt & > > logger -p local0.3 -f /tmp/lines5.txt & > > done > > > > sleep 1 > > ps aux | grep logger | wc -l > > > > if [[ "${foreverloop}" == "yes" ]]; then > > echo 'wait for loggers to exit' > > wait > > journalctl_pid=$(pidof journalctl) > > if [[ -z "${journalctl_pid}" ]]; then > > echo "journalctl dead, exit" > > exit 1 > > fi > > else > > exit 0 > > fi > > done > > > > > > 4) Wait for until the message "journalctl dead, exit" appears on the > console, this means journalclt exited. > > > > > > Rgds, > > --Ravi </blockquote></div>