[systemd-devel] Manual start of user@<uid>.service failed with permission denied
Mantas Mikulėnas
grawity at gmail.com
Wed Dec 13 07:08:23 UTC 2023
On Tue, Dec 12, 2023 at 6:15 PM Christopher Wong <Christopher.Wong at axis.com>
wrote:
> Hi Mantas,
>
>
>
> After user at 1001.service failed, it trigger the stopping process and
> become inactive.
>
Ah yeah, that makes sense, user-runtime-dir@ has StopWhenUnneeded=yes – so
of course after user at 1001 crashes you're not going to see anything mounted
anymore.
Could you try temporarily removing that option / setting it to 'no', just
to see what changes?
>
>
> ○ user-runtime-dir at 1001.service - User Runtime Directory /run/user/1001
>
> Loaded: loaded (/etc/systemd/system/user-runtime-dir at .service;
> static)
>
> Drop-In: /usr/lib/systemd/system/service.d
>
> └─10-axis.conf, 20-axis-sandbox.conf
>
> Active: inactive (dead) since Tue 2023-12-12 16:33:35 CET; 36min ago
>
> Duration: 315ms
>
> Docs: man:user at .service(5)
>
> Process: 16325 ExecStartPre=ls -la /run/user (code=exited,
> status=0/SUCCESS)
>
> Process: 16327 ExecStartPre=mount (code=exited, status=0/SUCCESS)
>
> Process: 16329 ExecStart=/usr/lib/systemd/systemd-user-runtime-dir
> start 1001 (code=exited, status=0/SUCCESS)
>
> Process: 16334 ExecStartPost=sleep 5 (code=exited, status=0/SUCCESS)
>
> Process: 16347 ExecStartPost=ls -la /run/user/1001 (code=exited,
> status=0/SUCCESS)
>
> Process: 16351 ExecStartPost=mount (code=exited, status=0/SUCCESS)
>
> Process: 16361 ExecStop=/usr/lib/systemd/systemd-user-runtime-dir
> stop 1001 (code=exited, status=0/SUCCESS)
>
> Main PID: 16329 (code=exited, status=0/SUCCESS)
>
> CPU: 48ms
>
>
>
> /etc/fstab don’t include anything on /run/user/1001 and there is no mount
> unit for run-user-1001.mount either.
>
>
>
> Best regards,
>
> Christopher Wong
>
>
>
>
>
> *From: *Mantas Mikulėnas <grawity at gmail.com>
> *Date: *Tuesday, 12 December 2023 at 17:05
> *To: *Christopher Wong <Christopher.Wong at axis.com>
> *Cc: *Systemd <systemd-devel at lists.freedesktop.org>
> *Subject: *Re: [systemd-devel] Manual start of user@<uid>.service failed
> with permission denied
>
> That sounds like it's getting immediately unmounted (or maybe not being
> mounted at all despite the program doing so).
>
>
>
> Does the user-runtime-dir service continue to show as "active" after this,
> or does it return to "inactive"?
>
>
>
> Does your /etc/fstab have any mentions of /run/user/1001? Or more
> generally, are there any run-user-1001.mount units? (If you 'systemctl
> status' this unit, does the status include a source path?)
>
>
>
> On Tue, Dec 12, 2023, 17:34 Christopher Wong <Christopher.Wong at axis.com>
> wrote:
>
> Hi Mantas,
>
>
>
> I currently have the following flow:
>
>
>
> 1. No /run/user/1001 directory
> 2. systemctl start user at 1001.service
> 3. systemd start user-runtime-dir at 1001.service which ends successfully.
> 4. The directory /run/user/1001 exists now, but is empty, owned by
> root with mode 0700
> 5. I don’t have findmnt on my system, so I used mount, but
> /run/user/1001 is not listed.
> 6. systemd start user at 1001.service which fails due to permission
> denied.
>
>
>
> I can’t explain why the /run/user/1001 is owned by root after
> user-runtime-dir at 1001.service successfully exited. I added some personal
> print in systemd code to ensure that the mount command returned success
> (r=0). Although, the mount was successful the command “mount” didn’t list
> it. In the list of mounts starting with /run I could only find these
> entries:
>
>
>
> Dec 12 16:19:35 host mount[14500]: tmpfs on /run type tmpfs
> (rw,nosuid,nodev,mode=755)
>
> Dec 12 16:19:35 host mount[14500]: tmpfs on /run/credentials type tmpfs
> (ro,nosuid,nodev,noexec,mode=755)
>
> Dec 12 16:19:35 host mount[14500]: tmpfs on /run/systemd/incoming type
> tmpfs (ro,nosuid,nodev,mode=755)
>
>
>
> If I do a chown of the directory in user at 1001.service then it works
>
>
>
> root at host:/run/user# ls -la 1001
>
> drwx------ 3 ida root 80 Dec 12 16:19 .
>
> drwxr-xr-x 3 root root 60 Dec 12 16:19 ..
>
> srw-rw-rw- 1 ida ssh-user 0 Dec 12 16:19 bus
>
> drwxr-xr-x 5 ida ssh-user 140 Dec 12 16:19 systemd
>
>
>
> The ”mount” command don’t list /run/user/1001 for the successful case
> either.
>
>
>
> Best regards,
>
> Christopher Wong
>
>
>
>
>
> *From: *Mantas Mikulėnas <grawity at gmail.com>
> *Date: *Monday, 11 December 2023 at 17:56
> *To: *Christopher Wong <Christopher.Wong at axis.com>
> *Cc: *Systemd <systemd-devel at lists.freedesktop.org>
> *Subject: *Re: [systemd-devel] Manual start of user@<uid>.service failed
> with permission denied
>
> On Mon, Dec 11, 2023, 17:28 Christopher Wong <Christopher.Wong at axis.com>
> wrote:
>
> Hi Mantas,
>
>
>
> I have added ExecStartPre to user at .service to run “id” and “ls -la”:
>
>
>
> Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Will mount
> /run/user/1001 owned by 1001:118
>
> Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Mounting tmpfs
> (tmpfs) on /run/user/1001 (MS_NOSUID|MS_NODEV
> "mode=0700,uid=1001,gid=118,size=99426304,nr_inodes=24274")...
>
> Dec 11 15:50:34 host systemd[1]: Finished User Runtime Directory
> /run/user/1001.
>
> Dec 11 15:50:34 host systemd[1]: Starting User Manager for UID 1001...
>
> Dec 11 15:50:34 host id[40291]: uid=1001(ida) gid=118(ssh-users)
> groups=118(ssh-users),236(systemd-journal)
>
> Dec 11 15:50:34 host ls[40293]: drwxr-xr-x 3 root root
> 60 Dec 11 15:50 .
>
> Dec 11 15:50:34 host ls[40293]: drwxr-xr-x 98 root root
> 2120 Dec 11 15:30 ..
>
> Dec 11 15:50:34 host ls[40293]: drwx------ 2 root root
> 40 Dec 11 15:50 1001
>
> Dec 11 15:50:34 host systemd[40294]: systemd 254.7-2-g9edc143 running in
> user mode for user 1001/ida. (-PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK
> +SECCOMP +GCRYPT +GNUTLS +OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2
> -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT
> -QRENCODE -TPM2 +BZIP2 -LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON -UTMP
> -SYSVINIT default-hierarchy=unified)
>
>
>
> The /run/user/1001 belongs to root with mode 0700. Should this belong to
> root?
>
> No, it should be owned by UID 1001 (though still mode 0700).
>
> Is it because I manually start user at 1001.service as root?
>
> Which user started the .service is usually not important, all services get
> a "fresh" environment that's fully described by the unit file.
>
>
>
> So even if you did 'systemctl start' as root, the unit has User=%i so the
> instance parameter tells it which UID to run as, so will be running as UID
> 1001. Likewise user-runtime-dir at 1001 will get the UID for the mount from
> its instance name (you can see that the "Mounting tmpfs" message has the
> correct information).
>
> However, after user-runtime-dir at 1001.service has finished it startup,
> the user at 1001.service is started as uid=1001 and therefore can’t create
> any directories under /run/user/1001. Resulting in user at 1001.service
> failed to start.
>
>
>
> If I add “ExecStartPre=+chown %i /run/user/%i” to user at .service then it
> works! But I am unsure if this is really the way fix this.
>
>
>
> So far, it sounds like the directory is being created *by something else*
> before user-runtime-dir@ is even invoked.
>
>
>
> Try adding the same "-/bin/ls -lad /run/user/%i" as both ExecStartPre and
> ExecStartPost of user-runtime-dir@ (and maybe even a findmnt). If the
> directory already exists during ExecStartPre, start looking for other
> services or cronjobs, or tmpfiles.d configs, or 'su' invocations, which may
> cause it to be created.
>
>
>
> There might also be something that chowns it to root *after* it was
> created correctly. If you actually see the tmpfs mount in 'findmnt' or in
> 'mount', but it's owned by root despite having uid=1001 in its mount
> options, something has chowned it...or your tmpfs feature is broken.
>
>
>
> If you don't see it in findmnt at all, even after user-runtime-dir has
> succeeded – either the mount failed quietly, or... something (like systemd
> itself) has quietly unmounted it.
>
>
>
>
>
> Regarding the testing, I have done both restart of everything and manual,
> but the result is the same. Now that I have the
> “Environment=XDG_RUNTIME_DIR=/run/user/%i” I no longer need to do
> “systemctl set-environment …”
>
>
>
> Thank you for taking your time!
>
>
>
> Best regards,
>
> Christopher Wong
>
>
>
>
>
> *From: *Mantas Mikulėnas <grawity at gmail.com>
> *Date: *Friday, 8 December 2023 at 21:53
> *To: *Christopher Wong <Christopher.Wong at axis.com>
> *Cc: *Systemd <systemd-devel at lists.freedesktop.org>
> *Subject: *Re: [systemd-devel] Manual start of user@<uid>.service failed
> with permission denied
>
> On Fri, Dec 8, 2023 at 6:53 PM Christopher Wong <Christopher.Wong at axis.com>
> wrote:
>
> Hi Mantas,
>
>
>
> I have from your suggestion done the following:
>
>
>
> Putting the below in user at .service
>
>
>
> [Service]
>
> ...
>
> Environment=XDG_RUNTIME_DIR=/run/user/%i
>
> Environment=SYSTEMD_LOG_LEVEL=debug
>
>
>
> Putting the below in user-runtime-dir at .service
>
>
>
> [Service]
>
> ...
>
> Environment=SYSTEMD_LOG_LEVEL=debug
>
>
>
> Then I have disabled the global set-log-level debug (if this is also
> required, please let me know).
>
>
>
> Unlike set-environment that's not global, it only affects pid1.
>
>
>
>
>
> What I can see from the logs is that user-runtime-dir at 1001.service seems
> to be started and mount /run/user/1001, but addition creation of directory
> under this mount is getting permission denied.
>
>
>
> Dec 08 17:33:29 host systemd[1]: Created slice User Slice of UID 1001.
>
> Dec 08 17:33:29 host systemd[1]: Starting User Runtime Directory
> /run/user/1001...
>
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing
> state UNSET -> OPENING
>
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: sd-bus: starting bus
> by connecting to /run/dbus/system_bus_socket...
>
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing
> state OPENING -> AUTHENTICATING
>
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing
> state AUTHENTICATING -> HELLO
>
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Sent message
> type=method_call sender=n/a destination=org.freedesktop.DBus
> path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello
> cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
>
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Got message
> type=method_return sender=org.freedesktop.DBus destination=:1.2536 path=n/a
> interface=n/a member=n/a cookie=1 reply_cookie=1 signature=s
> error-name=n/a error-message=n/a
>
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing
> state HELLO -> RUNNING
>
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Sent message
> type=method_call sender=n/a destination=org.freedesktop.login1
> path=/org/freedesktop/login1 interface=org.freedesktop.DBus.Properties
> member=Get cookie=2 reply_cookie=0 signature=ss error-name=n/a
> error-message=n/a
>
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Got message
> type=method_return sender=:1.323 destination=:1.2536 path=n/a interface=n/a
> member=n/a cookie=15 reply_cookie=2 signature=v error-name=n/a
> error-message=n/a
>
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Sent message
> type=method_call sender=n/a destination=org.freedesktop.login1
> path=/org/freedesktop/login1 interface=org.freedesktop.DBus.Properties
> member=Get cookie=3 reply_cookie=0 signature=ss error-name=n/a
> error-message=n/a
>
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Got message
> type=method_return sender=:1.323 destination=:1.2536 path=n/a interface=n/a
> member=n/a cookie=16 reply_cookie=3 signature=v error-name=n/a
> error-message=n/a
>
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing
> state RUNNING -> CLOSED
>
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Will mount
> /run/user/1001 owned by 1001:118
>
> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Mounting tmpfs
> (tmpfs) on /run/user/1001 (MS_NOSUID|MS_NODEV
> "mode=0700,uid=1001,gid=118,size=99426304,nr_inodes=24274")...
>
> Dec 08 17:33:29 host systemd[1]: Finished User Runtime Directory
> /run/user/1001.
>
> Dec 08 17:33:29 host systemd[1]: Starting User Manager for UID 1001...
>
> Dec 08 17:33:29 host systemd[36280]: systemd 254.7-2-g9edc143 running in
> user mode for user 1001/ida. (-PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK
> +SECCOMP +GCRYPT +GNUTLS +OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2
> -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT
> -QRENCODE -TPM2 +BZIP2 -LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON -UTMP
> -SYSVINIT default-hierarchy=unified)
>
> Dec 08 17:33:29 host systemd[36280]: Failed to create
> '/run/user/1001/systemd/inaccessible', ignoring: Permission denied
>
> Dec 08 17:33:29 host systemd[36280]: Failed to create
> '/run/user/1001/systemd/inaccessible/reg', ignoring: Permission denied
>
> Dec 08 17:33:29 host systemd[36280]: Failed to create
> '/run/user/1001/systemd/inaccessible/dir', ignoring: Permission denied
>
> Dec 08 17:33:29 host systemd[36280]: Failed to create
> '/run/user/1001/systemd/inaccessible/fifo', ignoring: Permission denied
>
> Dec 08 17:33:29 host systemd[36280]: Failed to create
> '/run/user/1001/systemd/inaccessible/sock', ignoring: Permission denied
>
> Dec 08 17:33:29 host systemd[36280]: Failed to create
> '/run/user/1001/systemd/inaccessible/chr', ignoring: Permission denied
>
> Dec 08 17:33:29 host systemd[36280]: Failed to create
> '/run/user/1001/systemd/inaccessible/blk', ignoring: Permission denied
>
>
>
> What's the ownership of /run/user/1001 and /run/user/1001/systemd after
> all of this?
>
>
>
> Are you rebooting between tests or just manually starting it?
>
>
>
> My current guess is that due to the earlier `systemctl set-environment`,
> some *other* thing that's running as root inherited the /run/user/1001 path
> and created root-owned directories there? That's the issue with setting
> global environment, it needs to be unset afterwards...
>
>
> --
>
> Mantas Mikulėnas
>
>
--
Mantas Mikulėnas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20231213/2cc06105/attachment-0001.htm>
More information about the systemd-devel
mailing list