[systemd-devel] Logs from a service is not showing up in journalctl but showing up in syslog

[Silvio Knizek]

Am Donnerstag, den 25.07.2019, 15:55 +0530 schrieb Debraj Manna:
> I have unit file which looks like below. I am seeing some of the echo
> are
> showing up in syslog but not in journalctl. Can someone let me know
> what is
> going on?
> systemd version 229 running on Ubuntu 16.
>
> [Unit]
> Description=Kafka Service
>
> [Service]
> Type=simple
> Environment=KAFKA_HOME=/home/ubuntu/deploy/kafka
> Environment=LIB_DIR=/var/lib/kafka
> Environment=LOG_DIR=/var/log/kafka
> Environment=TEMP_DIR=/home/ubuntu/tmp
> Environment=TOOLS_JAR=/home/ubuntu/build-target/common-utils/tools-
> 0.001-SNAPSHOT.jar
> Environment=MIN_DATA_PARTITION_FREE_SPACE_PCT=10
> Environment=MIN_DATA_PARTITION_FREE_SPACE_GB=10
> Environment=DATA_PARTITION_NAME=/var
>
> ExecStartPre=-/bin/mkdir -p /var/log/kafka
> ExecStartPre=-/bin/chown -R ubuntu:ubuntu /var/log/kafka
> ExecStartPre=-/bin/mkdir -p /var/lib/kafka/kafka-logs
> ExecStartPre=-/bin/chown -R ubuntu:ubuntu /var/lib/kafka/kafka-logs
> ExecStartPre=-/bin/rm -f /var/log/kafka/kafka-logs/.lock
> ExecStartPre=-/bin/mkdir -p /home/ubuntu/tmp
> ExecStartPre=-/bin/chown -R ubuntu:ubuntu /home/ubuntu/tmp
> ExecStartPre=-/bin/chmod -R 775 /home/ubuntu/tmp
> ExecStartPre=-/bin/su ubuntu -c
> "/home/ubuntu/build-target/kafka/kafka-systemd-prestart.sh"
> ExecStart=/bin/su ubuntu -c
> "/home/ubuntu/build-target/kafka/kafka-systemd-health.sh"
> ExecStopPost=-/bin/bash
> /home/ubuntu/build-target/kafka/kafka-systemd-poststop.sh
> RestartSec=2s
> Restart=always
> LimitNOFILE=65535
> KillSignal=SIGTERM
> SendSIGKILL=no
> SuccessExitStatus=1 143
>
> [Install]
> WantedBy=multi-user.target
>
> kafka-systemd-prestart.sh looks like below
>
> echo "[`date +"%h %d %Y %H:%M:%S-%3N"`] Starting kafka..."
> timeout --signal=sigkill 600s java -cp "$TOOLS_JAR"
> com.vnera.tools.kafka.KafkaIndexValidator "$LIB_DIR/kafka-logs"
> "$KAFKA_HOME/config/server.properties" true
> broker_id=`sudo grep -F "broker.id"
> $KAFKA_HOME/config/server.properties | awk -F '=' '{print $2}'`
> zookeeper_list=`sudo grep -F "zookeeper.connect="
> $KAFKA_HOME/config/server.properties | awk -F '=' '{print $2}'`
> echo "attempting removal of broker id $broker_id"
>
> used_pct=`df ${DATA_PARTITION_NAME} --output=pcent | grep -v Use |
> grep -o '[^ ]*[^ %]'`
> free_pct=$(expr 100 - $used_pct)
> free_gb=`df -h ${DATA_PARTITION_NAME} --output=avail --block-size G |
> grep -v Avail | grep -o '[^ ]*[^ G]'`
> echo "[`date +"%h %d %Y %H:%M:%S-%3N"`] partition
> ${DATA_PARTITION_NAME} free% $free_pct required%
> ${MIN_DATA_PARTITION_FREE_SPACE_PCT} freegb ${free_gb} required
> ${MIN_DATA_PARTITION_FREE_SPACE_GB}"
>
> # Some other code
>
> kafka-systemd-poststop.sh looks like below
>
> -------------------
>
> echo "[`date +"%h %d %Y %H:%M:%S-%3N"`] Exiting kafka..."
> cmd="ps -ef | grep -v grep | grep kafkaServer"
>
> # Some other code
>
> echo "completed exiting kafka"
>
> Doing sudo journalctl -u kafka.service looks like below
>
> Jul 25 07:41:39 platform2 systemd[1]: Started Kafka Service.
> Jul 25 07:41:39 platform2 su[39160]: Successful su for ubuntu by root
> Jul 25 07:41:39 platform2 su[39160]: + ??? root:ubuntu
> Jul 25 07:41:39 platform2 su[39160]: pam_unix(su:session): session
> opened for user ubuntu by (uid=0)
> Jul 25 07:41:40 platform2 bash[39192]: [Jul 25 2019 07:41:40-572]
> Exiting kafka...
>
> I am not seeing some of the echo from kafka-systemd-prestart.sh in
> journatl but I am seeing those logs in syslog
>
> Jul 25 10:17:03 platform2 su[38464]: WatchedEvent state:SyncConnected
> type:None path:null
> Jul 25 10:17:03 platform2 su[38464]: Node does not exist:
> /brokers/ids/2
> Jul 25 10:17:03 platform2 su[38464]: [Jul 25 2019 10:17:03-343]
> partition /var free% 9 required% 10 freegb 134 required 10
> Jul 25 10:17:03 platform2 su[38464]: [Jul 25 2019 10:17:03-344]
> Sufficient disk space is not available, sleeping for 60 seconds
> before
> exiting...

Hi,

first of all, take a look at man:tmpfiles.d to replace the whole
mkdir/chmod ExecStartPre= stuff.
Second, don't use »su« in .service. It breaks stuff by creating a new
cgroup hierarchy because it's run through pam. Use User= instead.
With both this changes your shell scripts shouldn't be necessary at all
and than everything should land in the journal.
Please don't re-invent the stuff systemd is already providing.

BR
Silvio