[systemd-devel] Restart sequence: "systemctl restart rsyslog.service syslog.socket" often/sometimes fails

Wed Feb 4 02:13:52 PST 2015

First: Please let me know if this is an inappropriate place to ask this

# systemctl restart syslog.socket rsyslog.service

seems to always work. But

# systemctl restart  rsyslog.service syslog.socket

, the opposite sequence, seems to sometimes fail (details below). I'm
wondering: Why is that?

And if I have a list of 20 PATTERNs, how do I devise the magical
pattern that ALWAYS works?

man systemctl[1] seems not to mention anything about sequence of PATTERN...s.

Depending on what hardware I try it occurs always, often, sometimes or
rarely. But trying it e.g. 1000 times seems to always trigger it at
sometimes. (A little scary that it doesn't always produce the same
result!!!)

BACKGROUND
============
I'm not quite certain that there is a need to restart syslog.socket
too. Let me know if that is the case.

But the question is still broader: How is the sequence of PATTERN...s
relevant for restart and how do I create a "good" sequence (even if we
omit syslog.socket).

Our app uses a number of system services (apache2, syslog-ng, mysql,
rabbitmq-server and some more). We have a randomly ordered list of
them. (Note we're actually using syslog-ng but the same ordering issue
also applies for rsyslog as shown above)

When we restore from a backup, we stop these services, put restored
data in place and start them again. Because of debian issue #751744:
"dh-systemd: postinst snippets should stop foo.socket during upgrades
too", syslog.socket is also in our list or it sometimes starts up
before we're done.

When we change configuration files, we restart that same list of
"names". So also syslog.socket.

DETAILS
=======
root at capmon:~# journalctl -xn --show-cursor -l --no-pager | tail -n 1
-- cursor: s=7cdc228fff4d49ada9fbebdfaf72e357;i=7a3;b=916402c8a5c747938b040a03c16b0e60;m=9e7556e8;t=50e404b5811ad;x=59fdfb3b30d4bdfa

root at capmon:~# systemctl restart  rsyslog.service syslog.socket
Job for syslog.socket failed. See 'systemctl status syslog.socket' and
'journalctl -xn' for details.
A dependency job for rsyslog.service failed. See 'journalctl -xn' for details.

root at capmon:~# journalctl -xn
--after-cursor='s=7cdc228fff4d49ada9fbebdfaf72e357;i=7a3;b=916402c8a5c747938b040a03c16b0e60;m=9e7556e8;t=50e404b5811ad;x=59fdfb3b30d4bdfa'
-l --no-pager
-- Logs begin at Wed 2015-02-04 10:18:51 CET, end at Wed 2015-02-04
11:06:42 CET. --
Feb 04 11:06:42 capmon systemd[1]: Stopping System Logging Service...
-- Subject: Unit rsyslog.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit rsyslog.service has begun shutting down.
Feb 04 11:06:42 capmon systemd[1]: Starting Syslog Socket.
-- Subject: Unit syslog.socket has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit syslog.socket has begun starting up.
Feb 04 11:06:42 capmon systemd[1]: Listening on Syslog Socket.
-- Subject: Unit syslog.socket has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit syslog.socket has finished starting up.
-- 
-- The start-up result is done.
Feb 04 11:06:42 capmon systemd[1]: Starting System Logging Service...
-- Subject: Unit rsyslog.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit rsyslog.service has begun starting up.
Feb 04 11:06:42 capmon systemd[1]: Stopping Syslog Socket.
-- Subject: Unit syslog.socket has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit syslog.socket has begun shutting down.
Feb 04 11:06:42 capmon systemd[1]: Starting Syslog Socket.
-- Subject: Unit syslog.socket has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit syslog.socket has begun starting up.
Feb 04 11:06:42 capmon systemd[1]: Socket service rsyslog.service
already active, refusing.
Feb 04 11:06:42 capmon systemd[1]: Failed to listen on Syslog Socket.
-- Subject: Unit syslog.socket has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit syslog.socket has failed.
-- 
-- The result is failed.
Feb 04 11:06:42 capmon systemd[1]: Dependency failed for System Logging Service.
-- Subject: Unit rsyslog.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit rsyslog.service has failed.
-- 
-- The result is dependency.

root at capmon:~# systemctl status syslog.socket -l
syslog.socket - Syslog Socket
   Loaded: loaded (/lib/systemd/system/syslog.socket; static)
   Active: inactive (dead) since Wed 2015-02-04 11:06:42 CET; 2min 58s ago
     Docs: man:systemd.special(7)
           http://www.freedesktop.org/wiki/Software/systemd/syslog
   Listen: /run/systemd/journal/syslog (Datagram)

Feb 04 11:06:42 capmon systemd[1]: Starting Syslog Socket.
Feb 04 11:06:42 capmon systemd[1]: Listening on Syslog Socket.
Feb 04 11:06:42 capmon systemd[1]: Stopping Syslog Socket.
Feb 04 11:06:42 capmon systemd[1]: Starting Syslog Socket.
Feb 04 11:06:42 capmon systemd[1]: Socket service rsyslog.service
already active, refusing.
Feb 04 11:06:42 capmon systemd[1]: Failed to listen on Syslog Socket.

# It looks to me like systemd should be able to establish the correct
restarts equence
root at capmon:~# systemctl list-dependencies  rsyslog.service | head -n2
rsyslog.service
|-syslog.socket

ENVIRONMENT
============
We have a debian testing/jessie based system that is more complicated,
but the above seems very easy to reproduce:

Install a minimum fresh system with the Debian jessie Beta 2 installer[2]
Run the commands above (in a for loop with many iterations to ensure
that it occurs)

# dpkg-query -W | egrep 'systemd|syslog' | sort
libsystemd0:amd64    215-10
libsystemd-daemon0:amd64    208-8
libsystemd-journal0:amd64    208-8
libsystemd-login0:amd64    208-8
rsyslog    8.4.2-1
systemd    208-8
systemd-sysv    208-8

# lsb_release -a
No LSB modules are available.
Distributor ID:    Debian
Description:    Debian GNU/Linux testing/unstable
Release:    testing/unstable
Codename:    n/a

LINKS
=====
1: http://www.freedesktop.org/software/systemd/man/systemctl.html
2: http://cdimage.debian.org/cdimage/jessie_di_beta_2/amd64/iso-cd/debian-jessie-DI-b2-amd64-netinst.iso
3: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=751744

-- 
Peter Valdemar Mørch
http://www.morch.com