<div dir="ltr"><div style="color:rgb(0,0,0);font-family:-webkit-standard"><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif"> </p></div><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span lang="EN-US">Hi,</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">Bug / Issue: </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">Under heavy logs entering the systemd-journald, journalctl Exits with one of the following error messages:</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">Failed to iterate through journal: Bad message</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">Failed to get realtime timestamp: Bad message</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">Failed to get monotonic timestamp: Bad message</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">The systemd is at version 243</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">Is this an Upstream bug which was fixed.?</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">N.B:- </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">As of now, not planning to upgrade to latest systemd version(247) for some legacy reasons.</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">Repro steps:-</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">1)Change the following 2 options in the file "/etc/systemd/journald.conf" from</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">RateLimitIntervalSec=1s</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">RateLimitBurst=2000</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">  to</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">RateLimitIntervalSec=0</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">RateLimitBurst=0</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">to turn off rate limiting, and then kill the "systemd-journald" process so that the new one </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">will be spawned and picks up the new jounald.conf</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">2) Start ‘journalctl’ with this command:   </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">  "/bin/journalctl -o json --output-fields=_SOURCE_REALTIME_TIMESTAMP,__REALTIME_TIMESTAMP,SYSLOG_IDENTIFIER,_SYSTEMD_UNIT,SYSLOG_FACILITY,MESSAGE,PRIORITY -f --no-tail > /dev/null 2>&1 &"</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">3)Create following bash script and run it as "test.sh 100 1"</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">--------------------</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">#! /bin/bash</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">if [[ ! -e /tmp/lines1.txt ]]; then</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">    for i in $(seq 1 50001); do echo "abc1-$i" >> /tmp/lines1.txt; done</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">    for i in $(seq 1 50001); do echo "abc2-$i" >> /tmp/lines2.txt; done</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">    for i in $(seq 1 50001); do echo "abc3-$i" >> /tmp/lines3.txt; done</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">    for i in $(seq 1 50001); do echo "abc4-$i" >> /tmp/lines4.txt; done</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">    for i in $(seq 1 50001); do echo "abc5-$i" >> /tmp/lines5.txt; done</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">fi</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">loggernum=${1}</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">if [[ -z ${loggernum} ]]; then</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">    loggernum=10</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">fi</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">loopnum=$((loggernum/5+1))</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">foreverloop='no'</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">if [[ -n "$2" ]]; then</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">    foreverloop='yes'</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">fi</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">while [[ 1 ]]; do</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">    for i in $(seq 1 ${loopnum}); do</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">        logger -p local0.3 -f /tmp/lines1.txt &</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">        logger -p local0.3 -f /tmp/lines2.txt &</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">        logger -p local0.3 -f /tmp/lines3.txt &</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">        logger -p local0.3 -f /tmp/lines4.txt &</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">        logger -p local0.3 -f /tmp/lines5.txt &</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">    done</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">    sleep 1</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">    ps aux | grep logger | wc -l</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">    if [[ "${foreverloop}" == "yes" ]]; then</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">        echo 'wait for loggers to exit'</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">        wait</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">        journalctl_pid=$(pidof journalctl)</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">        if [[ -z "${journalctl_pid}" ]]; then</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">            echo "journalctl dead, exit"</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">            exit 1</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">        fi</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">    else</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">        exit 0</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">    fi</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">done</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">4) Wait for until the message "journalctl dead, exit" appears on the console, this means journalclt exited. </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue""> </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">Rgds, </span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:10pt;font-family:"Helvetica Neue"">--Ravi</span></p><p class="MsoNormal" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;color:rgb(0,0,0)"> </p></div>