Troubleshooting timedatectl and hostnamectl failed to activate service: timed out

[Sean Caron]

Sorry, perhaps apparmor is not completely disabled, but I don't think it's
enforcing. I tried shutting it off completely with:

systemctl stop apparmor

And that doesn't seem to have made a difference.

Best,

Sean


On Thu, Dec 14, 2023 at 11:57 AM Sean Caron <scaron at umich.edu> wrote:

> Hi Mantas,
>
> Thanks for the suggestions! I took a look and I'm seeing entries like the
> following in the logs:
>
> Starting Hostname Service...
> systemd-hostnamed.service: Failed to set up mount namespacing:
> /run/systemd/unit-root/: Invalid argument
> systemd-hostnamed.service: Failed at step NAMESPACE spawning
> /lib/systemd/systemd-hostnamed: Invalid argument
> systemd-hostnamed.service: Main process exited, code=exited,
> status=226/NAMESPACE
> systemd-hostnamed.service: Failed with result 'exit-code'.
> Failed to start Hostname Service.
>
> Starting Time & Date Service...
> systemd-timedated.service: Failed to set up mount namespacing:
> /run/systemd/unit-root/: Invalid argument
> systemd-timedated.service: Failed at step NAMESPACE spawning
> /lib/systemd/systemd-timedated: Invalid argument
> systemd-timedated.service: Main process exited, code=exited,
> status=226/NAMESPACE
> systemd-timedated.service: Failed with result 'exit-code'.
> Failed to start Time & Date Service.
>
> Apparmor is disabled on all of our systems.
>
> The /run/systemd/unit-root directory exists on both working and nonworking
> systems and the ownership and permissions are identical on working and
> nonworking systems.
>
> I'm unfortunately not very conversant with everything that systemd does
> behind the scenes with this namespacing stuff. Does this raise any obvious
> flags? Any ideas for how I could further debug and/or resolve this would be
> so greatly appreciated!
>
> Best,
>
> Sean
>
> On Wed, Dec 13, 2023 at 1:22 PM Mantas Mikulėnas <grawity at gmail.com>
> wrote:
>
>> Activation is not client-side, it's handled automatically by dbus-daemon
>> – which either spawns the service directly or starts it as a systemd
>> service.
>>
>> In this case, check whether your logs show systemd-hostnamed.service
>> attempting to start; either it fails to start (missing libraries?
>> Apparmor?) or dbus-daemon fails to contact systemd (pid1 crashed?).
>>
>> On Wed, Dec 13, 2023, 19:45 Sean Caron <scaron at umich.edu> wrote:
>>
>>> Hi everyone,
>>>
>>> I'm on Ubuntu 20.04 LTS, kernel version 5.4.0-163-generic, systemd 245
>>> (245.4-4ubuntu3.22).
>>>
>>> I have some systems where I am receiving the following error messages
>>> when people attempt to use timedatectl or hostnamectl:
>>>
>>>
>>> Failed to query server: Failed to activate service
>>> 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms)
>>>
>>> Failed to query system properties: Failed to activate service
>>> 'org.freedesktop.hostname1': timed out (service_start_timeout=25000ms)
>>>
>>>
>>> I tried setting SYSTEMD_LOG_LEVEL=debug and rerunning the commands and
>>> it didn't really give me anything useful for determining the root cause of
>>> the issue. Here's an example of that output for timedatectl status:
>>>
>>>
>>> Bus n/a: changing state UNSET → OPENING
>>> Bus n/a: changing state OPENING → AUTHENTICATING
>>> Bus n/a: changing state AUTHENTICATING → HELLO
>>> Sent message type=method_call sender=n/a
>>> destination=org.freedesktop.DBus path=/org/freedesktop/DBus
>>> interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0
>>> signature=n/a error-name=n/a error-message=n/a
>>> Got message type=method_return sender=org.freedesktop.DBus
>>> destination=:1.15318 path=n/a interface=n/a member=n/a cookie=1
>>> reply_cookie=1 signature=s error-name=n/a error-message=n/a
>>> Bus n/a: changing state HELLO → RUNNING
>>> Sent message type=method_call sender=n/a
>>> destination=org.freedesktop.timedate1 path=/org/freedesktop/timedate1
>>> interface=org.freedesktop.DBus.Properties member=GetAll cookie=2
>>> reply_cookie=0 signature=s error-name=n/a error-message=n/a
>>> Got message type=error sender=org.freedesktop.DBus destination=:1.15318
>>> path=n/a interface=n/a member=n/a cookie=3 reply_cookie=2 signature=s
>>> error-name=org.freedesktop.DBus.Error.TimedOut error-message=Failed to
>>> activate service 'org.freedesktop.timedate1': timed out
>>> (service_start_timeout=25000ms)
>>> Failed to query server: Failed to activate service
>>> 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms)
>>> Bus n/a: changing state RUNNING → CLOSED
>>>
>>>
>>> I read that sometimes these issues can be caused by filesystem
>>> permissions on subdirectories in /var such as /var/tmp or /var/lib/systemd
>>> but I checked these and compared against a working system and I don't see
>>> any obvious differences.
>>>
>>> I have tried using strace on timedatectl and hostnamectl to try and see
>>> what's hanging things up but that hasn't really provided any fruitful
>>> direction, either.
>>>
>>> I didn't really know this was occurring until an end user reported it to
>>> me so I don't necessarily know how long the issue has been occurring or
>>> have a change in mind that could have broken things. I'm not sure if the
>>> upgrade from Ubuntu 18 to Ubuntu 20 broke it, or if some security
>>> configuration broke it. Or perhaps there is a missing dependency package on
>>> the broken systems?
>>>
>>> Could anyone out there please provide a little bit more guidance on how
>>> I might troubleshoot this and determine the root cause of the issue? I
>>> really hate to bother folks here but I'm feeling stuck.
>>>
>>> Thank you!
>>>
>>> Sean
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20231214/fdab0485/attachment-0001.htm>