[systemd-devel] systemd's connections to /run/systemd/private ?

[Zbigniew Jędrzejewski-Szmek]

On Tue, Jul 02, 2019 at 09:57:44AM -0400, Brian Reichert wrote:
> At $JOB, on some of our SLES12 boxes, our logs are getting swamped
> with messages saying:
> 
>   "Too many concurrent connections, refusing"

Please always specify the systemd version in use. We're not all SLES
users, and even if we were, I assume that there might be different
package versions over time.

>   # ss -x | grep /run/systemd/private | wc -l
>   4015

/run/systemd/private is used by systemctl and other systemd utilities
when running as root. Those connections are expected to be short-lived.
Generally, on a normal machine "ss -x | grep /run/systemd/private | wc -l"
is expected to yield 0 or a very low number transiently.

> But, despite the almost 4k connections, 'ss' shows that there are
> no connected peers:
> 
>   # ss -x | grep /run/systemd/private | grep -v -e '* 0' | wc -l
>   0

Interesting. ss output is not documented at all from what I can see,
but indeed '* 0' seems to indicate that. It is possible that systemd
has a leak and is not closing the private bus connections properly.

> When we first spin up a new SLES12 host with our custom services,
> the number of connections to /run/systemd/private numbers in the
> mere hundreds. 
That sounds wrong already. Please figure out what those connections
are. I'm afraid that you might have to do some debugging on your
own, since this issue doesn't seem easily reproducible.

(I installed systemd with CONNECTIONS_MAX set to 10, and I can easily
saturate the number of available connections with
  for i in {1..11}; do systemctl status '*' & sleep 0.5; kill -STOP $!;done
As soon as I allow the processes to continue or kill them, the connection
count goes down. They never show up with '* 0'.)

> Is my guess about CONNECTIONS_MAX's relationship to /run/systemd/private
> correct?

Yes. The number is hardcoded because it's expected to be "large
enough". The connection count shouldn't be more than "a few" or maybe
a dozen at any time.

> I have a hypothesis that this may be some resource leak in systemd,
> but I've not found a way to test that.

Once you figure out what is creating the connection, it would be useful
to attach strace to pid 1 and see what is happening there.

Zbyszek