[systemd-devel] Logs from a service is not showing up in journalctl but showing up in syslog

[Mantas Mikulėnas]

On Thu, Jul 25, 2019 at 1:26 PM Debraj Manna <subharaj.manna at gmail.com>
wrote:

> I have unit file which looks like below. I am seeing some of the echo are
> showing up in syslog but not in journalctl. Can someone let me know what is
> going on?
> systemd version 229 running on Ubuntu 16.
>
> [Unit]
> Description=Kafka Service
>
> [Service]
> Type=simple
> Environment=KAFKA_HOME=/home/ubuntu/deploy/kafka
> Environment=LIB_DIR=/var/lib/kafka
> Environment=LOG_DIR=/var/log/kafka
> Environment=TEMP_DIR=/home/ubuntu/tmp
>
> Environment=TOOLS_JAR=/home/ubuntu/build-target/common-utils/tools-0.001-SNAPSHOT.jar
> Environment=MIN_DATA_PARTITION_FREE_SPACE_PCT=10
> Environment=MIN_DATA_PARTITION_FREE_SPACE_GB=10
> Environment=DATA_PARTITION_NAME=/var
>
> ExecStartPre=-/bin/mkdir -p /var/log/kafka
> ExecStartPre=-/bin/chown -R ubuntu:ubuntu /var/log/kafka
> ExecStartPre=-/bin/mkdir -p /var/lib/kafka/kafka-logs
> ExecStartPre=-/bin/chown -R ubuntu:ubuntu /var/lib/kafka/kafka-logs
> ExecStartPre=-/bin/rm -f /var/log/kafka/kafka-logs/.lock
> ExecStartPre=-/bin/mkdir -p /home/ubuntu/tmp
> ExecStartPre=-/bin/chown -R ubuntu:ubuntu /home/ubuntu/tmp
> ExecStartPre=-/bin/chmod -R 775 /home/ubuntu/tmp
> ExecStartPre=-/bin/su ubuntu -c
> "/home/ubuntu/build-target/kafka/kafka-systemd-prestart.sh"
> ExecStart=/bin/su ubuntu -c
> "/home/ubuntu/build-target/kafka/kafka-systemd-health.sh"
>
> [...]

> Doing sudo journalctl -u kafka.service looks like below
>
> Jul 25 07:41:39 platform2 systemd[1]: Started Kafka Service.
> Jul 25 07:41:39 platform2 su[39160]: Successful su for ubuntu by root
> Jul 25 07:41:39 platform2 su[39160]: + ??? root:ubuntu
> Jul 25 07:41:39 platform2 su[39160]: pam_unix(su:session): session opened for user ubuntu by (uid=0)
> Jul 25 07:41:40 platform2 bash[39192]: [Jul 25 2019 07:41:40-572] Exiting kafka...
>
> I am not seeing some of the echo from kafka-systemd-prestart.sh in journatl but I am seeing those logs in syslog
>
> Jul 25 10:17:03 platform2 su[38464]: WatchedEvent state:SyncConnected type:None path:null
> Jul 25 10:17:03 platform2 su[38464]: Node does not exist: /brokers/ids/2
> Jul 25 10:17:03 platform2 su[38464]: [Jul 25 2019 10:17:03-343] partition /var free% 9 required% 10 freegb 134 required 10
> Jul 25 10:17:03 platform2 su[38464]: [Jul 25 2019 10:17:03-344] Sufficient disk space is not available, sleeping for 60 seconds before exiting...
>
>
Take a look at `journalctl -o verbose SYSLOG_IDENTIFIER=su _PID=38464`.

I suspect the messages *are* in the journal, just not tagged with
UNIT=kafka.service anymore. In some distros, `su` is actually configured to
call pam_systemd and set up a new systemd-logind session – when this
happens, the process is moved out of kafka.service into a user session
scope, and its syslog messages are grouped accordingly.

Consider replacing `su` with `runuser`, or indeed with systemd's [Service]
User= option.

-- 
Mantas Mikulėnas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20190725/1055bc01/attachment-0001.html>