Troubleshooting timedatectl and hostnamectl failed to activate service: timed out

[Sean Caron]

It appears that I can fix this by editing the unit file and changing:

ProtectSystem=strict

to:

ProtectSystem=full

I'm not sure why that is but a resolution is good enough for me. Following
up on my own thread in case this helps someone else in the future.

Best,

Sean


On Thu, Dec 14, 2023 at 12:02 PM Sean Caron <scaron at umich.edu> wrote:

> Sorry, perhaps apparmor is not completely disabled, but I don't think it's
> enforcing. I tried shutting it off completely with:
>
> systemctl stop apparmor
>
> And that doesn't seem to have made a difference.
>
> Best,
>
> Sean
>
>
> On Thu, Dec 14, 2023 at 11:57 AM Sean Caron <scaron at umich.edu> wrote:
>
>> Hi Mantas,
>>
>> Thanks for the suggestions! I took a look and I'm seeing entries like the
>> following in the logs:
>>
>> Starting Hostname Service...
>> systemd-hostnamed.service: Failed to set up mount namespacing:
>> /run/systemd/unit-root/: Invalid argument
>> systemd-hostnamed.service: Failed at step NAMESPACE spawning
>> /lib/systemd/systemd-hostnamed: Invalid argument
>> systemd-hostnamed.service: Main process exited, code=exited,
>> status=226/NAMESPACE
>> systemd-hostnamed.service: Failed with result 'exit-code'.
>> Failed to start Hostname Service.
>>
>> Starting Time & Date Service...
>> systemd-timedated.service: Failed to set up mount namespacing:
>> /run/systemd/unit-root/: Invalid argument
>> systemd-timedated.service: Failed at step NAMESPACE spawning
>> /lib/systemd/systemd-timedated: Invalid argument
>> systemd-timedated.service: Main process exited, code=exited,
>> status=226/NAMESPACE
>> systemd-timedated.service: Failed with result 'exit-code'.
>> Failed to start Time & Date Service.
>>
>> Apparmor is disabled on all of our systems.
>>
>> The /run/systemd/unit-root directory exists on both working and
>> nonworking systems and the ownership and permissions are identical on
>> working and nonworking systems.
>>
>> I'm unfortunately not very conversant with everything that systemd does
>> behind the scenes with this namespacing stuff. Does this raise any obvious
>> flags? Any ideas for how I could further debug and/or resolve this would be
>> so greatly appreciated!
>>
>> Best,
>>
>> Sean
>>
>> On Wed, Dec 13, 2023 at 1:22 PM Mantas Mikulėnas <grawity at gmail.com>
>> wrote:
>>
>>> Activation is not client-side, it's handled automatically by dbus-daemon
>>> – which either spawns the service directly or starts it as a systemd
>>> service.
>>>
>>> In this case, check whether your logs show systemd-hostnamed.service
>>> attempting to start; either it fails to start (missing libraries?
>>> Apparmor?) or dbus-daemon fails to contact systemd (pid1 crashed?).
>>>
>>> On Wed, Dec 13, 2023, 19:45 Sean Caron <scaron at umich.edu> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I'm on Ubuntu 20.04 LTS, kernel version 5.4.0-163-generic, systemd 245
>>>> (245.4-4ubuntu3.22).
>>>>
>>>> I have some systems where I am receiving the following error messages
>>>> when people attempt to use timedatectl or hostnamectl:
>>>>
>>>>
>>>> Failed to query server: Failed to activate service
>>>> 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms)
>>>>
>>>> Failed to query system properties: Failed to activate service
>>>> 'org.freedesktop.hostname1': timed out (service_start_timeout=25000ms)
>>>>
>>>>
>>>> I tried setting SYSTEMD_LOG_LEVEL=debug and rerunning the commands and
>>>> it didn't really give me anything useful for determining the root cause of
>>>> the issue. Here's an example of that output for timedatectl status:
>>>>
>>>>
>>>> Bus n/a: changing state UNSET → OPENING
>>>> Bus n/a: changing state OPENING → AUTHENTICATING
>>>> Bus n/a: changing state AUTHENTICATING → HELLO
>>>> Sent message type=method_call sender=n/a
>>>> destination=org.freedesktop.DBus path=/org/freedesktop/DBus
>>>> interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0
>>>> signature=n/a error-name=n/a error-message=n/a
>>>> Got message type=method_return sender=org.freedesktop.DBus
>>>> destination=:1.15318 path=n/a interface=n/a member=n/a cookie=1
>>>> reply_cookie=1 signature=s error-name=n/a error-message=n/a
>>>> Bus n/a: changing state HELLO → RUNNING
>>>> Sent message type=method_call sender=n/a
>>>> destination=org.freedesktop.timedate1 path=/org/freedesktop/timedate1
>>>> interface=org.freedesktop.DBus.Properties member=GetAll cookie=2
>>>> reply_cookie=0 signature=s error-name=n/a error-message=n/a
>>>> Got message type=error sender=org.freedesktop.DBus destination=:1.15318
>>>> path=n/a interface=n/a member=n/a cookie=3 reply_cookie=2 signature=s
>>>> error-name=org.freedesktop.DBus.Error.TimedOut error-message=Failed to
>>>> activate service 'org.freedesktop.timedate1': timed out
>>>> (service_start_timeout=25000ms)
>>>> Failed to query server: Failed to activate service
>>>> 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms)
>>>> Bus n/a: changing state RUNNING → CLOSED
>>>>
>>>>
>>>> I read that sometimes these issues can be caused by filesystem
>>>> permissions on subdirectories in /var such as /var/tmp or /var/lib/systemd
>>>> but I checked these and compared against a working system and I don't see
>>>> any obvious differences.
>>>>
>>>> I have tried using strace on timedatectl and hostnamectl to try and see
>>>> what's hanging things up but that hasn't really provided any fruitful
>>>> direction, either.
>>>>
>>>> I didn't really know this was occurring until an end user reported it
>>>> to me so I don't necessarily know how long the issue has been occurring or
>>>> have a change in mind that could have broken things. I'm not sure if the
>>>> upgrade from Ubuntu 18 to Ubuntu 20 broke it, or if some security
>>>> configuration broke it. Or perhaps there is a missing dependency package on
>>>> the broken systems?
>>>>
>>>> Could anyone out there please provide a little bit more guidance on how
>>>> I might troubleshoot this and determine the root cause of the issue? I
>>>> really hate to bother folks here but I'm feeling stuck.
>>>>
>>>> Thank you!
>>>>
>>>> Sean
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20231214/2fb2bdf5/attachment.htm>