[systemd-devel] [PATCH 3/3] core: support Distribute=n to distribute to n SO_REUSEPORT workers

[Lennart Poettering]

On Thu, 14.11.13 19:31, Zbigniew Jędrzejewski-Szmek (zbyszek at in.waw.pl) wrote:

> 
> On Thu, Nov 14, 2013 at 07:10:51PM +0100, Lennart Poettering wrote:
> > When the first connection comes in we'd spawn as many instances as
> > configured in Distribute=
> Hm, that seems like a big penalty. Why not instead:
> - when the first connection comes in, start one worker, keep listening
> - when the second connection comes in, start one worker, keep listening
> ...
> - when the n-th connection comes in, start one worker, stop listening
> 
> This way at least we don't have more workers than connections, and
> it staggers the launching of workers a bit, avoiding an overload.

Well, I don't see how we could make this work, neither with SO_REUSEPORT
nor with simple duplicated sockets. After all, in this case systemd
doesn't accept the connections, it just watches the original listening
fd for the first time it gets POLLIN on it. That's all. From that there
is no way to determine how many connections are currently going on,
i.e. how many connections other processes which share the fd have going
on.

If SO_REUSEPORT is used, then I'd expect PID 1 to hand the listening
socket it used itself to the first instance it spawned plus a new
socket that is bound to the same address to the second, and so on for
all others. Now, if PID 1 keeps watching that original fd, it will get a
wakeup only when the kernel decides to deliver an incoming connection to
the fd the first instance is using, and I doubt that is a particularly
useful information. If SO_RESUEPORT is not used, then I'd expect PID 1
to hand the listening socket to all instances. If it then kept watching
it, then it will get even worse information, it will in the worst case
wake up with every incoming connection, and in the best case miss a
number of them, and again wthout any chance to determine how many
incoming connections there are...

The only thing we could do is to parse /proc/net/tcp and count how many
connections are active bound to the same local address/port. But yikes,
that'd be ugly and inefficient.

Maybe one day the kernel als SO_GETCONCURRENT os so, which would tell us
how many connection sockets are bound to the same local address/port as
the socket we'd call this on is. Only then we could do such load
management...

Lennart

-- 
Lennart Poettering, Red Hat