[systemd-devel] Question: systemd-resolved.service delayed almost 90s start after systemd-sysctl.service
Alien Kong
kqleno at gmail.com
Wed May 21 14:00:12 UTC 2025
Hello,
This problem is a low-probability problem and is not easy to reproduce.
The boot.svg information of the most recent problem is attached. As can be
seen from the figure, the startup time of each service is normal, but it
inexplicably waits for nearly 90 seconds before systemd-resolved.service
starts.
Please help me find out how to troubleshoot this problem. It would be best
if you can give some troubleshooting suggestions. Thank you~
Best regards,
Alien Kong
Alien Kong <kqleno at gmail.com> 于2025年5月11日周日 13:50写道：
> Hi, Paul
>
> more information for reference.
> Looking forward to your new reply.Thank you!
>
> 1.We changed the default startup Timeout to 60s in
> /etc/systemd/system.conf.
> DefaultTimeoutStartSec=60s
>
> more system.conf information for reference.
>
> [Manager]
> #LogLevel=info
> #LogTarget=journal-or-kmsg
> #LogColor=yes
> #LogLocation=no
> #LogTime=no
> #DumpCore=yes
> #ShowStatus=yes
> #CrashChangeVT=no
> #CrashShell=no
> #CrashReboot=no
> #CtrlAltDelBurstAction=reboot-force
> #CPUAffinity=
> #NUMAPolicy=default
> #NUMAMask=
> #RuntimeWatchdogSec=off
> #RuntimeWatchdogPreSec=off
> #RuntimeWatchdogPreGovernor=
> #RebootWatchdogSec=10min
> #KExecWatchdogSec=off
> #WatchdogDevice=
> #CapabilityBoundingSet=
> #NoNewPrivileges=no
> #SystemCallArchitectures=
> #TimerSlackNSec=
> #StatusUnitFormat=combined
> #DefaultTimerAccuracySec=1min
> #DefaultStandardOutput=journal
> #DefaultStandardError=inherit
> DefaultTimeoutStartSec=60s
> #DefaultTimeoutStopSec=90s
> #DefaultTimeoutAbortSec=
> #DefaultDeviceTimeoutSec=90s
> #DefaultRestartSec=100ms
> #DefaultStartLimitIntervalSec=10s
> #DefaultStartLimitBurst=5
> #DefaultEnvironment=
> #DefaultCPUAccounting=yes
> #DefaultIOAccounting=no
> #DefaultIPAccounting=no
> #DefaultMemoryAccounting=yes
> #DefaultTasksAccounting=yes
> #DefaultTasksMax=15%
> #DefaultLimitCPU=
> #DefaultLimitFSIZE=
> #DefaultLimitDATA=
> #DefaultLimitSTACK=
> #DefaultLimitCORE=
> #DefaultLimitRSS=
> #DefaultLimitNOFILE=1024:524288
> #DefaultLimitAS=
> #DefaultLimitNPROC=
> #DefaultLimitMEMLOCK=8M
> #DefaultLimitLOCKS=
> #DefaultLimitSIGPENDING=
> #DefaultLimitMSGQUEUE=
> #DefaultLimitNICE=
> #DefaultLimitRTPRIO=
> #DefaultLimitRTTIME=
> #DefaultMemoryPressureThresholdSec=200ms
> #DefaultMemoryPressureWatch=auto
> #DefaultOOMPolicy=stop
> #DefaultSmackProcessLabel=
> #ReloadLimitIntervalSec=
> #ReloadLimitBurst=
>
>
> 2. From the serial port log, the time jump occurred at [2025-05-09
> 10:38:33.325848]
>
> [2025-05-09 10:38:33.325848] [ [0;32m OK [0m] Finished
> [0;1;39msystemd-sysctl.service [0m.
> // Here is the issue,no more logs output.
> [2025-05-09 10:40:02.693116] Starting
> [0;1;39msystemd-resolved.service [0m...
>
>
> more logs for reference.
> [2025-05-09 10:38:32.350945] oem-config setup has already been
> successfully completed
> [2025-05-09 10:38:32.382031] [ [0;32m OK [0m] Created slice
> [0;1;39msystem-getty.slice [0m.
> [2025-05-09 10:38:32.383543] [ [0;32m OK [0m] Created slice
> [0;1;39msystem-modprobe.slice [0m.
> [2025-05-09 10:38:32.400835] [ [0;32m OK [0m] Created slice
> [0;1;39msystem-serial\x2dgetty.slice [0m.
> [2025-05-09 10:38:32.414724] [ [0;32m OK [0m] Created slice
> [0;1;39muser.slice [0m.
> [2025-05-09 10:38:32.415990] [ [0;32m OK [0m] Started
> [0;1;39msystemd-ask-password-console.path [0m.
> [2025-05-09 10:38:32.416951] [ [0;32m OK [0m] Started
> [0;1;39msystemd-ask-password-wall.path [0m.
> [2025-05-09 10:38:32.430862] [ [0;32m OK [0m] Set up automount
> [0;1;39mproc-sys-fs-binfmt_misc.automount [0m.
> [2025-05-09 10:38:32.432453] Expecting device
> [0;1;39mdev-tegra_hv_pm_ctl.device [0m...
> [2025-05-09 10:38:32.446737] Expecting device
> [0;1;39mdev-ttyUTC0.device [0m...
> [2025-05-09 10:38:32.447717] [ [0;32m OK [0m] Reached target
> [0;1;39mcryptsetup.target [0m.
> [2025-05-09 10:38:32.448909] [ [0;32m OK [0m] Reached target
> [0;1;39mintegritysetup.target [0m.
> [2025-05-09 10:38:32.464302] [ [0;32m OK [0m] Reached target
> [0;1;39mpaths.target [0m.
> [2025-05-09 10:38:32.465641] [ [0;32m OK [0m] Reached target
> [0;1;39mslices.target [0m.
> [2025-05-09 10:38:32.479145] [ [0;32m OK [0m] Reached target
> [0;1;39mswap.target [0m.
> [2025-05-09 10:38:32.480320] [ [0;32m OK [0m] Reached target
> [0;1;39mtime-set.target [0m.
> [2025-05-09 10:38:32.494210] [ [0;32m OK [0m] Reached target
> [0;1;39mveritysetup.target [0m.
> [2025-05-09 10:38:32.495803] [ [0;32m OK [0m] Listening on
> [0;1;39mrpcbind.socket [0m.
> [2025-05-09 10:38:32.497627] [ [0;32m OK [0m] Listening on
> [0;1;39msyslog.socket [0m.
> [2025-05-09 10:38:32.510369] [ [0;32m OK [0m] Listening on
> [0;1;39msystemd-initctl.socket [0m.
> [2025-05-09 10:38:32.511771] [ [0;32m OK [0m] Listening on
> [0;1;39msystemd-journald-dev-log.socket [0m.
> [2025-05-09 10:38:32.513002] [ [0;32m OK [0m] Listening on
> [0;1;39msystemd-journald.socket [0m.
> [2025-05-09 10:38:32.526432] [ [0;32m OK [0m] Listening on
> [0;1;39msystemd-networkd.socket [0m.
> [2025-05-09 10:38:32.526937] [ [0;32m OK [0m] Listening on
> [0;1;39msystemd-udevd-control.socket [0m.
> [2025-05-09 10:38:32.542763] [ [0;32m OK [0m] Listening on
> [0;1;39msystemd-udevd-kernel.socket [0m.
> [2025-05-09 10:38:32.543647] Mounting [0;1;39mdev-hugepages.mount
> [0m...
> [2025-05-09 10:38:32.544342] Mounting [0;1;39mdev-mqueue.mount
> [0m...
> [2025-05-09 10:38:32.558008] Mounting
> [0;1;39msys-kernel-debug.mount [0m...
> [2025-05-09 10:38:32.558631] Mounting
> [0;1;39msys-kernel-tracing.mount [0m...
> [2025-05-09 10:38:32.559014] Starting
> [0;1;39mkmod-static-nodes.service [0m...
> [2025-05-09 10:38:32.574281] Starting
> [0;1;39mmodprobe at configfs.service [0m...
> [2025-05-09 10:38:32.574994] Starting
> [0;1;39mmodprobe at dm_mod.service [0m...
> [2025-05-09 10:38:32.575620] Starting [0;1;39mmodprobe at drm.service
> [0m...
> [2025-05-09 10:38:32.591030] Starting
> [0;1;39mmodprobe at efi_pstore.service [0m...
> [2025-05-09 10:38:32.591795] Starting
> [0;1;39mmodprobe at fuse.service [0m...
> [2025-05-09 10:38:32.593200] Starting
> [0;1;39mmodprobe at loop.service [0m...
> [2025-05-09 10:38:32.606240] Starting
> [0;1;39msystemd-modules-load.service [0m...
> [2025-05-09 10:38:32.607147] Starting
> [0;1;39msystemd-remount-fs.service [0m...
> [2025-05-09 10:38:32.607995] Starting
> [0;1;39msystemd-udev-trigger.service [0m...
> [2025-05-09 10:38:32.623071] [ [0;32m OK [0m] Mounted
> [0;1;39mdev-hugepages.mount [0m.
> [2025-05-09 10:38:32.624229] [ [0;32m OK [0m] Mounted
> [0;1;39mdev-mqueue.mount [0m.
> [2025-05-09 10:38:32.625780] [ [0;32m OK [0m] Mounted
> [0;1;39msys-kernel-debug.mount [0m.
> [2025-05-09 10:38:32.638671] [ [0;32m OK [0m] Mounted
> [0;1;39msys-kernel-tracing.mount [0m.
> [2025-05-09 10:38:32.639655] [ [0;32m OK [0m] Finished
> [0;1;39mkmod-static-nodes.service [0m.
> [2025-05-09 10:38:32.654140] [ [0;32m OK [0m] Finished
> [0;1;39mmodprobe at configfs.service [0m.
> [2025-05-09 10:38:32.655989] [ [0;32m OK [0m] Finished
> [0;1;39mmodprobe at dm_mod.service [0m.
> [2025-05-09 10:38:32.657256] [ [0;32m OK [0m] Finished
> [0;1;39mmodprobe at drm.service [0m.
> [2025-05-09 10:38:32.670735] [ [0;32m OK [0m] Finished
> [0;1;39mmodprobe at efi_pstore.service [0m.
> [2025-05-09 10:38:32.673000] [ [0;32m OK [0m] Finished
> [0;1;39mmodprobe at fuse.service [0m.
> [2025-05-09 10:38:32.674100] [ [0;32m OK [0m] Finished
> [0;1;39mmodprobe at loop.service [0m.
> [2025-05-09 10:38:32.690802] [ [0;32m OK [0m] Finished
> [0;1;39msystemd-remount-fs.service [0m.
> [2025-05-09 10:38:32.692796] Mounting
> [0;1;39msys-fs-fuse-connections.mount [0m...
> [2025-05-09 10:38:32.702262] Mounting
> [0;1;39msys-kernel-config.mount [0m...
> [2025-05-09 10:38:32.703737] Starting
> [0;1;39msystemd-random-seed.service [0m...
> [2025-05-09 10:38:32.705370] Starting
> [0;1;39msystemd-tmpfiles-setup-dev-early.service [0m...
> [2025-05-09 10:38:32.719696] [ [0;32m OK [0m] Mounted
> [0;1;39msys-fs-fuse-connections.mount [0m.
> [2025-05-09 10:38:32.722370] [ [0;32m OK [0m] Mounted
> [0;1;39msys-kernel-config.mount [0m.
> [2025-05-09 10:38:32.724814] [ [0;32m OK [0m] Finished
> [0;1;39msystemd-random-seed.service [0m.
> [2025-05-09 10:38:32.735303] Starting
> [0;1;39mmodprobe at dm_mod.service [0m...
> [2025-05-09 10:38:32.736278] Starting
> [0;1;39mmodprobe at efi_pstore.service [0m...
> [2025-05-09 10:38:32.739990] Starting
> [0;1;39mmodprobe at loop.service [0m...
> [2025-05-09 10:38:32.750934] [ [0;32m OK [0m] Finished
> [0;1;39mmodprobe at dm_mod.service [0m.
> [2025-05-09 10:38:32.752393] [ [0;32m OK [0m] Finished
> [0;1;39mmodprobe at efi_pstore.service [0m.
> [2025-05-09 10:38:32.766126] [ [0;32m OK [0m] Finished
> [0;1;39mmodprobe at loop.service [0m.
> [2025-05-09 10:38:32.767311] [ [0;32m OK [0m] Finished
> [0;1;39msystemd-tmpfiles-setup-dev-early.service [0m.
> [2025-05-09 10:38:32.783233] Starting
> [0;1;39msystemd-tmpfiles-setup-dev.service [0m...
> [2025-05-09 10:38:32.786457] [ [0;32m OK [0m] Reached target
> [0;1;39mtime-sync.target [0m.
> [2025-05-09 10:38:32.799052] Starting
> [0;1;39msystemd-journald.service [0m...
> [2025-05-09 10:38:32.799957] [ [0;32m OK [0m] Started
> [0;1;39msystemd-journald.service [0m.
> [2025-05-09 10:38:32.813973] Starting
> [0;1;39msystemd-journal-flush.service [0m...
> [2025-05-09 10:38:32.815059] [ [0;32m OK [0m] Finished
> [0;1;39msystemd-udev-trigger.service [0m.
> [2025-05-09 10:38:32.816318] [ [0;32m OK [0m] Finished
> [0;1;39msystemd-tmpfiles-setup-dev.service [0m.
> [2025-05-09 10:38:32.830379] [ [0;32m OK [0m] Reached target
> [0;1;39mlocal-fs-pre.target [0m.
> [2025-05-09 10:38:32.832103] [ [0;32m OK [0m] Reached target
> [0;1;39mlocal-fs.target [0m.
> [2025-05-09 10:38:32.834105] [ [0;32m OK [0m] Listening on
> [0;1;39msystemd-sysext.socket [0m.
> [2025-05-09 10:38:32.847003] Starting
> [0;1;39msystemd-binfmt.service [0m...
> [2025-05-09 10:38:32.848522] Starting
> [0;1;39msystemd-udevd.service [0m...
> [2025-05-09 10:38:32.862854] Mounting
> [0;1;39mproc-sys-fs-binfmt_misc.mount [0m...
> [2025-05-09 10:38:32.863965] [ [0;32m OK [0m] Mounted
> [0;1;39mproc-sys-fs-binfmt_misc.mount [0m.
> [2025-05-09 10:38:32.878898] [ [0;32m OK [0m] Finished
> [0;1;39msystemd-binfmt.service [0m.
> [2025-05-09 10:38:32.879833] [ [0;32m OK [0m] Finished
> [0;1;39msystemd-journal-flush.service [0m.
> [2025-05-09 10:38:32.894091] Starting
> [0;1;39msystemd-tmpfiles-setup.service [0m...
> [2025-05-09 10:38:32.895352] [ [0;32m OK [0m] Started
> [0;1;39msystemd-udevd.service [0m.
> [2025-05-09 10:38:33.022039] [ [0;32m OK [0m] Finished
> [0;1;39msystemd-modules-load.service [0m.
> [2025-05-09 10:38:33.039129] [ [0;32m OK [0m] Finished
> [0;1;39msystemd-tmpfiles-setup.service [0m.
> [2025-05-09 10:38:33.165964] [ [0;32m OK [0m] Found device
> [0;1;39mdev-ttyUTC0.device [0m.
> [2025-05-09 10:38:33.229922] Mounting
> [0;1;39mrun-rpc_pipefs.mount [0m...
> [2025-05-09 10:38:33.246783] Starting [0;1;39mrpcbind.service
> [0m...
> [2025-05-09 10:38:33.247943] Starting
> [0;1;39msystemd-sysctl.service [0m...
> [2025-05-09 10:38:33.249473] Starting
> [0;1;39msystemd-update-utmp.service [0m...
> [2025-05-09 10:38:33.262237] [ [0;32m OK [0m] Mounted
> [0;1;39mrun-rpc_pipefs.mount [0m.
> [2025-05-09 10:38:33.263251] [ [0;32m OK [0m] Found device
> [0;1;39mdev-tegra_hv_pm_ctl.device [0m.
> [2025-05-09 10:38:33.265105] [ [0;32m OK [0m] Started
> [0;1;39mrpcbind.service [0m.
> [2025-05-09 10:38:33.278968] [ [0;32m OK [0m] Reached target
> [0;1;39mrpc_pipefs.target [0m.
> [2025-05-09 10:38:33.280378] [ [0;32m OK [0m] Reached target
> [0;1;39mrpcbind.target [0m.
> [2025-05-09 10:38:33.293863] [ [0;32m OK [0m] Reached target
> [0;1;39mnfs-client.target [0m.
> [2025-05-09 10:38:33.294915] [ [0;32m OK [0m] Reached target
> [0;1;39mremote-fs-pre.target [0m.
> [2025-05-09 10:38:33.296171] [ [0;32m OK [0m] Reached target
> [0;1;39mremote-fs.target [0m.
> [2025-05-09 10:38:33.310669] [ [0;32m OK [0m] Finished
> [0;1;39msystemd-update-utmp.service [0m.
> [2025-05-09 10:38:33.325848] [ [0;32m OK [0m] Finished
> [0;1;39msystemd-sysctl.service [0m.
> // Here is the issue.
> [2025-05-09 10:40:02.693116] Starting
> [0;1;39msystemd-resolved.service [0m...
> [2025-05-09 10:40:03.093112] [ [0;32m OK [0m] Started
> [0;1;39msystemd-resolved.service [0m.
> [2025-05-09 10:40:03.094834] [ [0;32m OK [0m] Reached target
> [0;1;39mnss-lookup.target [0m.
> [2025-05-09 10:40:03.110386] [ [0;32m OK [0m] Reached target
> [0;1;39msysinit.target [0m.
> [2025-05-09 10:40:03.111699] [ [0;32m OK [0m] Started
> [0;1;39mdpkg-db-backup.timer [0m.
> [2025-05-09 10:40:03.125272] [ [0;32m OK [0m] Started
> [0;1;39me2scrub_all.timer [0m.
> [2025-05-09 10:40:03.126731] [ [0;32m OK [0m] Started
> [0;1;39mfstrim.timer [0m.
> [2025-05-09 10:40:03.127872] [ [0;32m OK [0m] Started
> [0;1;39mlogrotate.timer [0m.
> [2025-05-09 10:40:03.141129] [ [0;32m OK [0m] Started
> [0;1;39mmotd-news.timer [0m.
> [2025-05-09 10:40:03.142378] [ [0;32m OK [0m] Started
> [0;1;39msystemd-tmpfiles-clean.timer [0m.
> [2025-05-09 10:40:03.144127] [ [0;32m OK [0m] Started
> [0;1;39mxm_logrotate.timer [0m.
> [2025-05-09 10:40:03.157836] [ [0;32m OK [0m] Reached target
> [0;1;39mtimers.target [0m.
> [2025-05-09 10:40:03.159283] [ [0;32m OK [0m] Listening on
> [0;1;39mdbus.socket [0m.
> [2025-05-09 10:40:03.160640] [ [0;32m OK [0m] Listening on
> [0;1;39mssh.socket [0m.
> [2025-05-09 10:40:03.173272] [ [0;32m OK [0m] Reached target
> [0;1;39msockets.target [0m.
> [2025-05-09 10:40:03.174836] [ [0;32m OK [0m] Reached target
> [0;1;39mbasic.target [0m.
> [2025-05-09 10:40:03.176052] Starting [0;1;39mdbus.service [0m...
> [2025-05-09 10:40:03.189270] Starting
> [0;1;39mdisable_power_features_linux.service [0m...
> [2025-05-09 10:40:03.190659] [ [0;32m OK [0m] Started
> [0;1;39mdmesg.service [0m.
> [2025-05-09 10:40:03.191302] Starting
> [0;1;39me2scrub_reap.service [0m...
>
> Best regards,
> Alien Kong
>
> Alien Kong <kqleno at gmail.com> 于2025年5月10日周六 22:55写道：
>
>> Hi, Paul
>>
>> Thank you for your reply.
>>
>> This is a low-probability, sporadic problem. Currently, there is no
>> `journalctl -b` information with `debug` on the command line.
>>
>> The unit information of the two services is as follows:
>> 1. systemctl cat systemd-resolved
>>
>> # /usr/lib/systemd/system/systemd-resolved.service
>> # SPDX-License-Identifier: LGPL-2.1-or-later
>> #
>> # This file is part of systemd.
>> #
>> # systemd is free software; you can redistribute it and/or modify it
>> # under the terms of the GNU Lesser General Public License as published
>> by
>> # the Free Software Foundation; either version 2.1 of the License, or
>> # (at your option) any later version.
>>
>> [Unit]
>> Description=Network Name Resolution
>> Documentation=man:systemd-resolved.service(8)
>> Documentation=man:org.freedesktop.resolve1(5)
>> Documentation=
>> https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
>> Documentation=
>> https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
>>
>> DefaultDependencies=no
>> After=systemd-sysctl.service systemd-sysusers.service
>> Before=sysinit.target network.target nss-lookup.target shutdown.target
>> initrd-switch-root.target
>> Conflicts=shutdown.target initrd-switch-root.target
>> Wants=nss-lookup.target
>>
>> [Service]
>> AmbientCapabilities=CAP_SETPCAP CAP_NET_RAW CAP_NET_BIND_SERVICE
>> BusName=org.freedesktop.resolve1
>> CapabilityBoundingSet=CAP_SETPCAP CAP_NET_RAW CAP_NET_BIND_SERVICE
>> ExecStart=!!/usr/lib/systemd/systemd-resolved
>> LockPersonality=yes
>> MemoryDenyWriteExecute=yes
>> NoNewPrivileges=yes
>> PrivateDevices=yes
>> PrivateTmp=yes
>> ProtectClock=yes
>> ProtectControlGroups=yes
>> ProtectHome=yes
>> ProtectKernelLogs=yes
>> ProtectKernelModules=yes
>> ProtectKernelTunables=yes
>> ProtectSystem=strict
>> Restart=always
>> RestartSec=0
>> RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6
>> RestrictNamespaces=yes
>> RestrictRealtime=yes
>> RestrictSUIDSGID=yes
>> RuntimeDirectory=systemd/resolve
>> RuntimeDirectoryPreserve=yes
>> SystemCallArchitectures=native
>> SystemCallErrorNumber=EPERM
>> SystemCallFilter=@system-service
>> Type=notify
>> User=systemd-resolve
>> ImportCredential=network.dns
>> ImportCredential=network.search_domains
>> WatchdogSec=3min
>>
>> [Install]
>> WantedBy=sysinit.target
>> Alias=dbus-org.freedesktop.resolve1.service
>> 2. systemctl cat systemd-sysctl.service
>> # /usr/lib/systemd/system/systemd-sysctl.service
>> # SPDX-License-Identifier: LGPL-2.1-or-later
>> #
>> # This file is part of systemd.
>> #
>> # systemd is free software; you can redistribute it and/or modify it
>> # under the terms of the GNU Lesser General Public License as published
>> by
>> # the Free Software Foundation; either version 2.1 of the License, or
>> # (at your option) any later version.
>>
>> [Unit]
>> Description=Apply Kernel Variables
>> Documentation=man:systemd-sysctl.service(8) man:sysctl.d(5)
>> DefaultDependencies=no
>> Conflicts=shutdown.target
>> After=systemd-modules-load.service
>> Before=sysinit.target shutdown.target
>> ConditionPathIsReadWrite=/proc/sys/net/
>>
>> [Service]
>> Type=oneshot
>> RemainAfterExit=yes
>> ExecStart=/usr/lib/systemd/systemd-sysctl
>> TimeoutSec=90s
>> ImportCredential=sysctl.*
>>
>> Best regards,
>> Alien Kong
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20250521/992149d7/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: boot.svg
Type: image/svg+xml
Size: 314263 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20250521/992149d7/attachment-0001.svg>
More information about the systemd-devel
mailing list