Troubleshooting timedatectl and hostnamectl failed to activate service: timed out

[Sean Caron]

Hi Mantas,

Thanks for the suggestions! I took a look and I'm seeing entries like the
following in the logs:

Starting Hostname Service...
systemd-hostnamed.service: Failed to set up mount namespacing:
/run/systemd/unit-root/: Invalid argument
systemd-hostnamed.service: Failed at step NAMESPACE spawning
/lib/systemd/systemd-hostnamed: Invalid argument
systemd-hostnamed.service: Main process exited, code=exited,
status=226/NAMESPACE
systemd-hostnamed.service: Failed with result 'exit-code'.
Failed to start Hostname Service.

Starting Time & Date Service...
systemd-timedated.service: Failed to set up mount namespacing:
/run/systemd/unit-root/: Invalid argument
systemd-timedated.service: Failed at step NAMESPACE spawning
/lib/systemd/systemd-timedated: Invalid argument
systemd-timedated.service: Main process exited, code=exited,
status=226/NAMESPACE
systemd-timedated.service: Failed with result 'exit-code'.
Failed to start Time & Date Service.

Apparmor is disabled on all of our systems.

The /run/systemd/unit-root directory exists on both working and nonworking
systems and the ownership and permissions are identical on working and
nonworking systems.

I'm unfortunately not very conversant with everything that systemd does
behind the scenes with this namespacing stuff. Does this raise any obvious
flags? Any ideas for how I could further debug and/or resolve this would be
so greatly appreciated!

Best,

Sean

On Wed, Dec 13, 2023 at 1:22 PM Mantas Mikulėnas <grawity at gmail.com> wrote:

> Activation is not client-side, it's handled automatically by dbus-daemon –
> which either spawns the service directly or starts it as a systemd service.
>
> In this case, check whether your logs show systemd-hostnamed.service
> attempting to start; either it fails to start (missing libraries?
> Apparmor?) or dbus-daemon fails to contact systemd (pid1 crashed?).
>
> On Wed, Dec 13, 2023, 19:45 Sean Caron <scaron at umich.edu> wrote:
>
>> Hi everyone,
>>
>> I'm on Ubuntu 20.04 LTS, kernel version 5.4.0-163-generic, systemd 245
>> (245.4-4ubuntu3.22).
>>
>> I have some systems where I am receiving the following error messages
>> when people attempt to use timedatectl or hostnamectl:
>>
>>
>> Failed to query server: Failed to activate service
>> 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms)
>>
>> Failed to query system properties: Failed to activate service
>> 'org.freedesktop.hostname1': timed out (service_start_timeout=25000ms)
>>
>>
>> I tried setting SYSTEMD_LOG_LEVEL=debug and rerunning the commands and it
>> didn't really give me anything useful for determining the root cause of the
>> issue. Here's an example of that output for timedatectl status:
>>
>>
>> Bus n/a: changing state UNSET → OPENING
>> Bus n/a: changing state OPENING → AUTHENTICATING
>> Bus n/a: changing state AUTHENTICATING → HELLO
>> Sent message type=method_call sender=n/a destination=org.freedesktop.DBus
>> path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello
>> cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
>> Got message type=method_return sender=org.freedesktop.DBus
>> destination=:1.15318 path=n/a interface=n/a member=n/a cookie=1
>> reply_cookie=1 signature=s error-name=n/a error-message=n/a
>> Bus n/a: changing state HELLO → RUNNING
>> Sent message type=method_call sender=n/a
>> destination=org.freedesktop.timedate1 path=/org/freedesktop/timedate1
>> interface=org.freedesktop.DBus.Properties member=GetAll cookie=2
>> reply_cookie=0 signature=s error-name=n/a error-message=n/a
>> Got message type=error sender=org.freedesktop.DBus destination=:1.15318
>> path=n/a interface=n/a member=n/a cookie=3 reply_cookie=2 signature=s
>> error-name=org.freedesktop.DBus.Error.TimedOut error-message=Failed to
>> activate service 'org.freedesktop.timedate1': timed out
>> (service_start_timeout=25000ms)
>> Failed to query server: Failed to activate service
>> 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms)
>> Bus n/a: changing state RUNNING → CLOSED
>>
>>
>> I read that sometimes these issues can be caused by filesystem
>> permissions on subdirectories in /var such as /var/tmp or /var/lib/systemd
>> but I checked these and compared against a working system and I don't see
>> any obvious differences.
>>
>> I have tried using strace on timedatectl and hostnamectl to try and see
>> what's hanging things up but that hasn't really provided any fruitful
>> direction, either.
>>
>> I didn't really know this was occurring until an end user reported it to
>> me so I don't necessarily know how long the issue has been occurring or
>> have a change in mind that could have broken things. I'm not sure if the
>> upgrade from Ubuntu 18 to Ubuntu 20 broke it, or if some security
>> configuration broke it. Or perhaps there is a missing dependency package on
>> the broken systems?
>>
>> Could anyone out there please provide a little bit more guidance on how I
>> might troubleshoot this and determine the root cause of the issue? I
>> really hate to bother folks here but I'm feeling stuck.
>>
>> Thank you!
>>
>> Sean
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20231214/29e814a6/attachment.htm>