<div dir="ltr">There is one instance per connection; they get cleaned up after a successful stop, but accumulate on failures.</div> It seems that your service unit depends on firewalld, and firewalld is masked (i.e. forbidden to start), so the service breaks due to dependency failure.Not sure where that dependency comes from – maybe you have a drop-in adding this dependency, try "systemctl cat commandsocket_57813@.service". That will show all files systemd reads, which might be more than one.(Or maybe your distro patched it in for _all_ sockets? Wouldn't be surprised.)But in general, this doesn't seem a very reliable check. Do you care about specific services, like HTTP? Then monitor *those* services instead. Do you use this as theft detection? Power outage watch? <div class="gmail_quote"><div dir="ltr">On Fri, May 5, 2017, 11:30 Robert Pilja <<a href="mailto:robert.pilja@1und1.de" target="_blank">robert.pilja@1und1.de</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div lang="DE" link="blue" vlink="purple"> <div class="m_1970690722920681119m_-8930679515489802774WordSection1"> Hello, i'm looking for help on debugging a systemd issue after upgrading our systems from RHEL 7.2 to 7.3 (systemd: 219-19.el7_2.9.ppc64 -> 219-30.el7_3.7.ppc64). We are using a custom socket unit which basically opens a tcp port and returns /bin/true to established connections. This unit provides an healthcheck/heartbeat mechanism used by our loadbalancer framework. For the last 1.5 years, this implementation worked flawless. Until now. Here is our config: /etc/systemd/system/commandsocket_57813@.service [Unit] Description=service for /bin/true ListenStream Port 57813 host present check [Service] ExecStart=/bin/true StandardOutput=socket /etc/systemd/system/commandsocket_57813.socket [Unit] Description=Port 57813 socket for host present check [Socket] ListenStream=57813 Accept=yes [Install] WantedBy=sockets.target --- journalctl: Apr 24 03:38:11 bscqs01.server.lan systemd[1]: Failed to start service for /bin/true ListenStream Port 57813 host present check. Apr 24 03:38:11 bscqs01.server.lan systemd[1]: Unit commandsocket_57813@65308-10.88.44.62:57813-10.88.44.27:51072.service entered failed state. Apr 24 03:38:11 bscqs01.server.lan systemd[1]: commandsocket_57813@65308-10.88.44.62:57813-10.88.44.27:51072.service failed. Apr 24 03:38:11 bscqs01.server.lan systemd[1]: Starting service for /bin/true ListenStream Port 57813 host present check... Apr 24 03:38:16 bscqs01.server.lan systemd[1]: Cannot add dependency job for unit firewalld.service, ignoring: Unit is masked. Apr 24 03:38:16 bscqs01.server.lan systemd[1]: commandsocket_57813@65309-10.88.44.62:57813-10.88.44.27:51161.service failed to run 'start' task: Transport endpoint is not connected Apr 24 03:38:16 bscqs01.server.lan systemd[1]: Failed to start service for /bin/true ListenStream Port 57813 host present check (<a href="http://10.88.44.27:51161" target="_blank">10.88.44.27:51161</a>). Apr 24 03:38:16 bscqs01.server.lan systemd[1]: Unit commandsocket_57813@65309-10.88.44.62:57813-10.88.44.27:51161.service entered failed state. Apr 24 03:38:16 bscqs01.server.lan systemd[1]: commandsocket_57813@65309-10.88.44.62:57813-10.88.44.27:51161.service failed. Apr 24 03:38:16 bscqs01.server.lan systemd[1]: Starting service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:51161)... Apr 24 03:38:26 bscqs01.server.lan systemd[1]: Cannot add dependency job for unit firewalld.service, ignoring: Unit is masked. Apr 24 03:38:26 bscqs01.server.lan systemd[1]: commandsocket_57813@65310-10.88.44.62:57813-10.88.44.27:51346.service failed to run 'start' task: Transport endpoint is not connected Apr 24 03:38:26 bscqs01.server.lan systemd[1]: Failed to start service for /bin/true ListenStream Port 57813 host present check (<a href="http://10.88.44.27:51346" target="_blank">10.88.44.27:51346</a>). Apr 24 03:38:26 bscqs01.server.lan systemd[1]: Unit commandsocket_57813@65310-10.88.44.62:57813-10.88.44.27:51346.service entered failed state. Apr 24 03:38:26 bscqs01.server.lan systemd[1]: commandsocket_57813@65310-10.88.44.62:57813-10.88.44.27:51346.service failed. Apr 24 03:38:26 bscqs01.server.lan systemd[1]: Starting service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:51346)... Apr 24 03:38:31 bscqs01.server.lan systemd[1]: Cannot add dependency job for unit firewalld.service, ignoring: Unit is masked. Apr 24 03:38:31 bscqs01.server.lan systemd[1]: commandsocket_57813@65311-10.88.44.62:57813-10.88.44.27:51435.service failed to run 'start' task: Transport endpoint is not connected -- systemctl start commandsocket_57813.socket systemctl status commandsocket_57813.socket ● commandsocket_57813.socket - Port 57813 socket for host present check Loaded: loaded (/etc/systemd/system/commandsocket_57813.socket; enabled; vendor preset: disabled) Active: active (listening) since Fri 2017-05-05 08:48:48 CEST; 5s ago Listen: [::]:57813 (Stream) Accepted: 65359; Connected: 0 May 05 08:48:48 bscqs01.server.lan systemd[1]: Listening on Port 57813 socket for host present check. May 05 08:48:48 bscqs01.server.lan systemd[1]: Starting Port 57813 socket for host present check. systemctl status commandsocket_57813.socket ● commandsocket_57813.socket - Port 57813 socket for host present check Loaded: loaded (/etc/systemd/system/commandsocket_57813.socket; enabled; vendor preset: disabled) Active: failed (Result: resources) since Fri 2017-05-05 08:48:54 CEST; 7s ago Listen: [::]:57813 (Stream) Accepted: 65359; Connected: 0 May 05 08:48:48 bscqs01.server.lan systemd[1]: Listening on Port 57813 socket for host present check. May 05 08:48:48 bscqs01.server.lan systemd[1]: Starting Port 57813 socket for host present check. May 05 08:48:54 bscqs01.server.lan systemd[1]: commandsocket_57813.socket failed to queue service startup job (Maybe the service file is missing or not a template unit?): Argument list too long May 05 08:48:54 bscqs01.server.lan systemd[1]: Unit commandsocket_57813.socket entered failed state. -- systemctl |grep commandsocket_57813: [...] ● commandsocket_57813@9991-10.88.44.62:57813-10.88.44.27:54933.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (<a href="http://10.88.44.27:54933" target="_blank">10.88.44.27:54933</a>) ● commandsocket_57813@9992-10.88.44.62:57813-10.88.44.27:55028.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (<a href="http://10.88.44.27:55028" target="_blank">10.88.44.27:55028</a>) ● commandsocket_57813@9993-10.88.44.62:57813-10.88.44.27:55119.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (<a href="http://10.88.44.27:55119" target="_blank">10.88.44.27:55119</a>) ● commandsocket_57813@9994-10.88.44.62:57813-10.88.44.27:55207.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (<a href="http://10.88.44.27:55207" target="_blank">10.88.44.27:55207</a>) ● commandsocket_57813@9995-10.88.44.62:57813-10.88.44.27:55302.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (<a href="http://10.88.44.27:55302" target="_blank">10.88.44.27:55302</a>) ● commandsocket_57813@9996-10.88.44.62:57813-10.88.44.27:55393.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (<a href="http://10.88.44.27:55393" target="_blank">10.88.44.27:55393</a>) ● commandsocket_57813@9997-10.88.44.62:57813-10.88.44.27:55481.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (<a href="http://10.88.44.27:55481" target="_blank">10.88.44.27:55481</a>) ● commandsocket_57813@9998-10.88.44.62:57813-10.88.44.27:55576.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (<a href="http://10.88.44.27:55576" target="_blank">10.88.44.27:55576</a>) ● commandsocket_57813@9999-10.88.44.62:57813-10.88.44.27:55667.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (<a href="http://10.88.44.27:55667" target="_blank">10.88.44.27:55667</a>) system-commandsocket_57813.slice loaded active active system-commandsocket_57813.slice ● commandsocket_57813.socket loaded failed failed Port 57813 socket for host present check systemctl |grep commandsocket_57813 | wc -l: 65361 Port number limit? --- Any idea on how to fix this problem? Do you think that our current unit implementation is a reasonable solution for providing healthchecks? Before systemd, we were using xinetd for that. Is this large number of additional entries (commandsocket_57813@9991-10.88.44.62:57813-10.88.44.27:54933.service etc.) normal? I have not seen them before the upgrade. Regards Robert </div> </div> _______________________________________________ systemd-devel mailing list <a href="mailto:systemd-devel@lists.freedesktop.org" target="_blank">systemd-devel@lists.freedesktop.org</a> <a href="https://lists.freedesktop.org/mailman/listinfo/systemd-devel" rel="noreferrer" target="_blank">https://lists.freedesktop.org/mailman/listinfo/systemd-devel</a> </blockquote></div><div dir="ltr">-- </div><div data-smartmail="gmail_signature">Mantas Mikulėnas <<a href="mailto:grawity@gmail.com">grawity@gmail.com</a>> Sent from my phone </div>