[systemd-devel] systemd's connections to /run/systemd/private ?

[Zbigniew Jędrzejewski-Szmek]

On Thu, Jul 11, 2019 at 10:08:43AM -0400, Brian Reichert wrote:
> On Wed, Jul 10, 2019 at 10:44:14PM +0000, Zbigniew J??drzejewski-Szmek wrote:
> > That's ancient... 228 was released almost four years ago.
> 
> That's the joy of using a commercial Linux distribution; they tend
> to be conservative about updates.  SLES may very well have backported
> fixes to the packaged version they maintain.
> 
> They may also have a newer version of a systemd RPM for us to take.
> 
> I'm looking for an efficient way to repro the symptoms, as to confirm
> whether a newer RPM solves this for us.
> 
> > > > > When we first spin up a new SLES12 host with our custom services,
> > > > > the number of connections to /run/systemd/private numbers in the
> > > > > mere hundreds. 
> > > 
> > > > That sounds wrong already. Please figure out what those connections
> > > > are. I'm afraid that you might have to do some debugging on your
> > > > own, since this issue doesn't seem easily reproducible.
> 
> Above, I cite a desire for reproducing the symptoms.  If you're
> confident that a newly-spun-up idle host should not hover at hundreds
> of connections, then hypothetically I could update the vendor-provided
> systemd RPM (if there is one), reboot, and see if the connection
> count is reduced.
> 
> > strace -p1 -e recvmsg,close,accept4,getsockname,getsockopt,sendmsg -s999
> >
> > yields the relevant info. In particular, the pid, uid, and guid of the
> > remote is shown. My approach would be to log this to some file, and
> > then see which fds remain, and then look up this fd in the log.
> > The recvmsg calls contain the serialized dbus calls, a bit messy but
> > understandable. E.g. 'systemctl show systemd-udevd' gives something
> > like this:
> 
> Thanks for such succinct feedback; I'll see what I can get from this.
> 
> In my prior email, I showed how some of the connections were
> hours/days old, even with no connecting peer.
> 
> Does that sound like expected behavior?

No, this shouldn't happen.

What I was trying to say, is that if you have the strace log, you
can figure out what created the stale connection and what the dbus
call was, and from all that info it should be fairly simply to figure
out what the calling command was. Once you have that, it'll be much
easier to reproduce the issue in controlled setting and look for the
fix.

Zbyszek