[systemd-devel] systemd-resolved: performance question

[Petr Menšík]

Hi Robert,

interesting. It seems resolved is not expecting so heavy usage. Consider 
other cache until this is fixed, unbound or dnsmasq might be a good choice.

Please create an issue for it on https://github.com/systemd/systemd. 
Especially when it can use the fact it is a deamon, it should be able to 
use notifications of /etc/resolv.conf symlink changes. It should not 
require stat of that file on each query. It definitely should save that 
when the resolve NSS plugin is the first one, even before files.

It should watch the file with some filesystem notification system. 
Especially when it already knows it is the service supposed to maintain 
that file. I don't understand why it is checking both resolv.conf and 
stub-resolv.conf. Only one of them should be under active use.

Even if it could not use filesystem monitoring, I guess it could check 
those files only once per second or so. Should not depend on number of 
done queries.

Please share your distribution and systemd version in it, it is unclear 
what version are we talking about. resolvectl status command output 
would be also useful.

On 3/14/23 19:25, Robert Ayrapetyan wrote:
> Hello, I'm using systemd-resolved on the server which performs a lot
> of DNS queries (~20K per second) and systemd-resolved helps a lot
> providing a cache:
>
> Cache
>    Current Cache Size: 263
>            Cache Hits: 30928976
>            Cache Misses: 2961
>
> However, systemd-resolved process almost constantly utilizes 100% of
> CPU (I don't have any other dns services like dnsmasq installed).
>
> strace shows this:
>
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>   48.73    0.966765           4    194141           stat
>   15.06    0.298804          12     24106           sendmsg
>   13.59    0.269703           8     32346     32346 openat
>    6.42    0.127381           5     24118           recvmsg
>    5.49    0.108829           4     24118           recvfrom
>    5.34    0.106037           4     24118           epoll_wait
>    5.33    0.105838           4     24118           gettid
>    0.01    0.000151           6        24           epoll_ctl
>
> And in particular for stat, it queries just 3 files in a loop:
>
> stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=738, ...}) = 0
> <0.000010>
> stat("/run/systemd/resolve/resolv.conf", {st_mode=S_IFREG|0644,
> st_size=624, ...}) = 0 <0.000009>
> stat("/run/systemd/resolve/stub-resolv.conf", {st_mode=S_IFREG|0644,
> st_size=738, ...}) = 0 <0.000013>
>
> My question is: why does it "stats" so often? Is this a result of my
> misconfiguration?
>
> /etc/systemd/resolved.conf:
> ...
> FallbackDNS=1.1.1.1 1.0.0.1
>
> /etc/nsswitch.conf:
> ...
> hosts:          resolve [!UNAVAIL=return] files dns
>
-- 
Petr Menšík
Software Engineer, RHEL
Red Hat, https://www.redhat.com/
PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB