<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-2022-jp"> <meta name="Generator" content="Microsoft Word 14 (filtered medium)"> <style></style> </head> <body lang="DE" link="blue" vlink="purple"> <div class="WordSection1"> Hello,<o:p></o:p> i'm looking for help on debugging a systemd issue after upgrading our systems from RHEL 7.2 to 7.3 (systemd: 219-19.el7_2.9.ppc64 -> 219-30.el7_3.7.ppc64).<o:p></o:p> <o:p> </o:p> We are using a custom socket unit which basically opens a tcp port and returns /bin/true to established connections.<o:p></o:p> This unit provides an healthcheck/heartbeat mechanism used by our loadbalancer framework.<o:p></o:p> For the last 1.5 years, this implementation worked flawless. Until now.<o:p></o:p> <o:p> </o:p> Here is our config:<o:p></o:p> <o:p> </o:p> /etc/systemd/system/commandsocket_57813@.service<o:p></o:p> [Unit]<o:p></o:p> Description=service for /bin/true ListenStream Port 57813 host present check<o:p></o:p> <o:p> </o:p> [Service]<o:p></o:p> ExecStart=/bin/true<o:p></o:p> StandardOutput=socket<o:p></o:p> <o:p></o:p> /etc/systemd/system/commandsocket_57813.socket<o:p></o:p> [Unit]<o:p></o:p> Description=Port 57813 socket for host present check<o:p></o:p> <o:p> </o:p> [Socket]<o:p></o:p> ListenStream=57813<o:p></o:p> Accept=yes<o:p></o:p> <o:p> </o:p> [Install]<o:p></o:p> WantedBy=sockets.target<o:p></o:p> <o:p></o:p> ---<o:p></o:p> <o:p> </o:p> journalctl:<o:p></o:p> <o:p> </o:p> Apr 24 03:38:11 bscqs01.server.lan systemd[1]: Failed to start service for /bin/true ListenStream Port 57813 host present check.<o:p></o:p> Apr 24 03:38:11 bscqs01.server.lan systemd[1]: Unit commandsocket_57813@65308-10.88.44.62:57813-10.88.44.27:51072.service entered failed state.<o:p></o:p> Apr 24 03:38:11 bscqs01.server.lan systemd[1]: commandsocket_57813@65308-10.88.44.62:57813-10.88.44.27:51072.service failed.<o:p></o:p> Apr 24 03:38:11 bscqs01.server.lan systemd[1]: Starting service for /bin/true ListenStream Port 57813 host present check...<o:p></o:p> Apr 24 03:38:16 bscqs01.server.lan systemd[1]: Cannot add dependency job for unit firewalld.service, ignoring: Unit is masked.<o:p></o:p> Apr 24 03:38:16 bscqs01.server.lan systemd[1]: commandsocket_57813@65309-10.88.44.62:57813-10.88.44.27:51161.service failed to run 'start' task: Transport endpoint is not connected<o:p></o:p> Apr 24 03:38:16 bscqs01.server.lan systemd[1]: Failed to start service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:51161).<o:p></o:p> Apr 24 03:38:16 bscqs01.server.lan systemd[1]: Unit commandsocket_57813@65309-10.88.44.62:57813-10.88.44.27:51161.service entered failed state.<o:p></o:p> Apr 24 03:38:16 bscqs01.server.lan systemd[1]: commandsocket_57813@65309-10.88.44.62:57813-10.88.44.27:51161.service failed.<o:p></o:p> Apr 24 03:38:16 bscqs01.server.lan systemd[1]: Starting service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:51161)...<o:p></o:p> Apr 24 03:38:26 bscqs01.server.lan systemd[1]: Cannot add dependency job for unit firewalld.service, ignoring: Unit is masked.<o:p></o:p> Apr 24 03:38:26 bscqs01.server.lan systemd[1]: commandsocket_57813@65310-10.88.44.62:57813-10.88.44.27:51346.service failed to run 'start' task: Transport endpoint is not connected<o:p></o:p> Apr 24 03:38:26 bscqs01.server.lan systemd[1]: Failed to start service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:51346).<o:p></o:p> Apr 24 03:38:26 bscqs01.server.lan systemd[1]: Unit commandsocket_57813@65310-10.88.44.62:57813-10.88.44.27:51346.service entered failed state.<o:p></o:p> Apr 24 03:38:26 bscqs01.server.lan systemd[1]: commandsocket_57813@65310-10.88.44.62:57813-10.88.44.27:51346.service failed.<o:p></o:p> Apr 24 03:38:26 bscqs01.server.lan systemd[1]: Starting service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:51346)...<o:p></o:p> Apr 24 03:38:31 bscqs01.server.lan systemd[1]: Cannot add dependency job for unit firewalld.service, ignoring: Unit is masked.<o:p></o:p> Apr 24 03:38:31 bscqs01.server.lan systemd[1]: commandsocket_57813@65311-10.88.44.62:57813-10.88.44.27:51435.service failed to run 'start' task: Transport endpoint is not connected<o:p></o:p> <o:p> </o:p> --<o:p></o:p> <o:p> </o:p> systemctl start commandsocket_57813.socket<o:p></o:p> <o:p> </o:p> systemctl status commandsocket_57813.socket<o:p></o:p> $B!|(B commandsocket_57813.socket - Port 57813 socket for host present check<o:p></o:p> Loaded: loaded (/etc/systemd/system/commandsocket_57813.socket; enabled; vendor preset: disabled)<o:p></o:p> Active: active (listening) since Fri 2017-05-05 08:48:48 CEST; 5s ago<o:p></o:p> Listen: [::]:57813 (Stream)<o:p></o:p> Accepted: 65359; Connected: 0<o:p></o:p> <o:p> </o:p> May 05 08:48:48 bscqs01.server.lan systemd[1]: Listening on Port 57813 socket for host present check.<o:p></o:p> May 05 08:48:48 bscqs01.server.lan systemd[1]: Starting Port 57813 socket for host present check.<o:p></o:p> <o:p> </o:p> systemctl status commandsocket_57813.socket<o:p></o:p> $B!|(B commandsocket_57813.socket - Port 57813 socket for host present check<o:p></o:p> Loaded: loaded (/etc/systemd/system/commandsocket_57813.socket; enabled; vendor preset: disabled)<o:p></o:p> Active: failed (Result: resources) since Fri 2017-05-05 08:48:54 CEST; 7s ago<o:p></o:p> Listen: [::]:57813 (Stream)<o:p></o:p> Accepted: 65359; Connected: 0<o:p></o:p> <o:p> </o:p> May 05 08:48:48 bscqs01.server.lan systemd[1]: Listening on Port 57813 socket for host present check.<o:p></o:p> May 05 08:48:48 bscqs01.server.lan systemd[1]: Starting Port 57813 socket for host present check.<o:p></o:p> May 05 08:48:54 bscqs01.server.lan systemd[1]: commandsocket_57813.socket failed to queue service startup job (Maybe the service file is missing or not a template unit?): Argument list too long<o:p></o:p> May 05 08:48:54 bscqs01.server.lan systemd[1]: Unit commandsocket_57813.socket entered failed state.<o:p></o:p> <o:p> </o:p> --<o:p></o:p> <o:p></o:p> systemctl |grep commandsocket_57813:<o:p></o:p> [...]<o:p></o:p> $B!|(B commandsocket_57813@9991-10.88.44.62:57813-10.88.44.27:54933.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:54933)<o:p></o:p> $B!|(B commandsocket_57813@9992-10.88.44.62:57813-10.88.44.27:55028.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:55028)<o:p></o:p> $B!|(B commandsocket_57813@9993-10.88.44.62:57813-10.88.44.27:55119.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:55119)<o:p></o:p> $B!|(B commandsocket_57813@9994-10.88.44.62:57813-10.88.44.27:55207.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:55207)<o:p></o:p> $B!|(B commandsocket_57813@9995-10.88.44.62:57813-10.88.44.27:55302.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:55302)<o:p></o:p> $B!|(B commandsocket_57813@9996-10.88.44.62:57813-10.88.44.27:55393.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:55393)<o:p></o:p> $B!|(B commandsocket_57813@9997-10.88.44.62:57813-10.88.44.27:55481.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:55481)<o:p></o:p> $B!|(B commandsocket_57813@9998-10.88.44.62:57813-10.88.44.27:55576.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:55576)<o:p></o:p> $B!|(B commandsocket_57813@9999-10.88.44.62:57813-10.88.44.27:55667.service loaded failed failed service for /bin/true ListenStream Port 57813 host present check (10.88.44.27:55667)<o:p></o:p> system-commandsocket_57813.slice loaded active active system-commandsocket_57813.slice<o:p></o:p> $B!|(B commandsocket_57813.socket loaded failed failed Port 57813 socket for host present check<o:p></o:p> <o:p> </o:p> systemctl |grep commandsocket_57813 | wc -l:<o:p></o:p> 65361<o:p></o:p> <o:p> </o:p> Port number limit?<o:p></o:p> <o:p> </o:p> ---<o:p></o:p> <o:p> </o:p> Any idea on how to fix this problem?<o:p></o:p> <o:p> </o:p> Do you think that our current unit implementation is a reasonable solution for providing healthchecks? Before systemd, we were using xinetd for that.<o:p></o:p> Is this large number of additional entries (commandsocket_57813@9991-10.88.44.62:57813-10.88.44.27:54933.service etc.) normal? I have not seen them before the upgrade.<o:p></o:p> <o:p> </o:p> Regards<o:p></o:p> Robert<o:p></o:p> </div> </body> </html>