[systemd-devel] Manual start of user@<uid>.service failed with permission denied
Christopher Wong
Christopher.Wong at axis.com
Wed Dec 13 12:19:13 UTC 2023
Hi Mantas,
I finally found out what was causing this mount issue. We have a global sandboxing configuration for all services, that implies PrivateNetwork=yes. According to manual this will imply PrivateMounts=yes. So, I added a drop-in configuration to user-runtime-dir at .service.d<mailto:user-runtime-dir at .service.d> which defines PrivateMounts=no. Now it works! 😊
Thank you very much for your support!
Best regards,
Christopher Wong
From: systemd-devel <systemd-devel-bounces at lists.freedesktop.org> on behalf of Christopher Wong <Christopher.Wong at axis.com>
Date: Wednesday, 13 December 2023 at 11:32
To: Mantas Mikulėnas <grawity at gmail.com>
Cc: Systemd <systemd-devel at lists.freedesktop.org>
Subject: Re: [systemd-devel] Manual start of user@<uid>.service failed with permission denied
Hi Mantas,
I ran your suggestion and here is the result:
root at host:/run/user# ls -la
drwxr-xr-x 2 root root 40 Dec 13 11:24 .
drwxr-xr-x 99 root root 2160 Dec 13 11:00 ..
root at host:/run/user# /usr/lib/systemd/systemd-user-runtime-dir start 1001
root at host:/run/user# ls -la
drwxr-xr-x 3 root root 60 Dec 13 11:25 .
drwxr-xr-x 99 root root 2160 Dec 13 11:00 ..
drwx------ 2 ida ssh-user 40 Dec 13 11:25 1001
root at host:/run/user# mount | grep /run
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=97096k,nr_inodes=24274,mode=700,uid=1001,gid=118)
This seems to be correct, and I could see /run/user/1001 listed using “mount”.
Thank you for your explaination! We have probably be running with a uncomplete setup of the mount before.
Best regards,
Christopher Wong
From: Mantas Mikulėnas <grawity at gmail.com>
Date: Wednesday, 13 December 2023 at 10:33
To: Christopher Wong <Christopher.Wong at axis.com>
Cc: Systemd <systemd-devel at lists.freedesktop.org>
Subject: Re: [systemd-devel] Manual start of user@<uid>.service failed with permission denied
On Wed, Dec 13, 2023 at 10:36 AM Christopher Wong <Christopher.Wong at axis.com<mailto:Christopher.Wong at axis.com>> wrote:
Hi Mantas,
I tried with StopWhenUnneeded=no in user-runtime-dir at .service<mailto:user-runtime-dir at .service>, then when user at 1001.service<mailto:user at 1001.service> fails the status of user-runtime-dir at .service<mailto:user-runtime-dir at .service> is active. At this state the directory /run/user/1001 is created, it is empty, owned by root. Running the mount command doesn’t show /run/user/1001.
Run the "/usr/lib/systemd/systemd-user-runtime-dir start 1001" manually and check whether the mounted filesystem is there afterwards.
If it's still not there, then run "mount -t tmpfs -o uid=1001,mode=0700 none /run/user/1001" and then check whether it stays mounted.
I have mentioned it before, but I want to point out that if I put “ExecStartPre=+chown %i /run/user/%i” in user at .service<mailto:user at .service> then the user at 1001.service<mailto:user at 1001.service> can be started manually. The mount command doesn’t show /run/user/1001 either, but since the service is started the path contains bus socket and systemd directory with content, which are the things I am after.
The main issue here is that /run/user/1001 is owned by root after user-runtime-dir at .service<mailto:user-runtime-dir at .service> has been exited successfully.
No, that's only a symptom of the main issue.
The current design that systemd implements is to have a user-specific tmpfs mounted at that location (for quota purposes), and so the underlying mountpoint is deliberately created as owned by root – its ownership is not changed because it's supposed to have a new filesystem mounted on top (which would make the mountpoint hidden and its ownership moot).
If you specifically want to *not* have an additional tmpfs there, then you can continue using the manual "ExecStartPre=chown" (or in fact you could replace the entire user-runtime-dir@ with a simpler one that only mkdirs and chowns), but in that case you shouldn't be saying that it's a systemd issue that it doesn't chown something that it was never meant to chown to begin with.
Best regards,
Christopher Wong
From: Mantas Mikulėnas <grawity at gmail.com<mailto:grawity at gmail.com>>
Date: Wednesday, 13 December 2023 at 08:08
To: Christopher Wong <Christopher.Wong at axis.com<mailto:Christopher.Wong at axis.com>>
Cc: Systemd <systemd-devel at lists.freedesktop.org<mailto:systemd-devel at lists.freedesktop.org>>
Subject: Re: [systemd-devel] Manual start of user@<uid>.service failed with permission denied
On Tue, Dec 12, 2023 at 6:15 PM Christopher Wong <Christopher.Wong at axis.com<mailto:Christopher.Wong at axis.com>> wrote:
Hi Mantas,
After user at 1001.service<mailto:user at 1001.service> failed, it trigger the stopping process and become inactive.
Ah yeah, that makes sense, user-runtime-dir@ has StopWhenUnneeded=yes – so of course after user at 1001 crashes you're not going to see anything mounted anymore.
Could you try temporarily removing that option / setting it to 'no', just to see what changes?
○ user-runtime-dir at 1001.service<mailto:user-runtime-dir at 1001.service> - User Runtime Directory /run/user/1001
Loaded: loaded (/etc/systemd/system/user-runtime-dir at .service<mailto:/etc/systemd/system/user-runtime-dir at .service>; static)
Drop-In: /usr/lib/systemd/system/service.d
└─10-axis.conf, 20-axis-sandbox.conf
Active: inactive (dead) since Tue 2023-12-12 16:33:35 CET; 36min ago
Duration: 315ms
Docs: man:user at .service(5)
Process: 16325 ExecStartPre=ls -la /run/user (code=exited, status=0/SUCCESS)
Process: 16327 ExecStartPre=mount (code=exited, status=0/SUCCESS)
Process: 16329 ExecStart=/usr/lib/systemd/systemd-user-runtime-dir start 1001 (code=exited, status=0/SUCCESS)
Process: 16334 ExecStartPost=sleep 5 (code=exited, status=0/SUCCESS)
Process: 16347 ExecStartPost=ls -la /run/user/1001 (code=exited, status=0/SUCCESS)
Process: 16351 ExecStartPost=mount (code=exited, status=0/SUCCESS)
Process: 16361 ExecStop=/usr/lib/systemd/systemd-user-runtime-dir stop 1001 (code=exited, status=0/SUCCESS)
Main PID: 16329 (code=exited, status=0/SUCCESS)
CPU: 48ms
/etc/fstab don’t include anything on /run/user/1001 and there is no mount unit for run-user-1001.mount either.
Best regards,
Christopher Wong
From: Mantas Mikulėnas <grawity at gmail.com<mailto:grawity at gmail.com>>
Date: Tuesday, 12 December 2023 at 17:05
To: Christopher Wong <Christopher.Wong at axis.com<mailto:Christopher.Wong at axis.com>>
Cc: Systemd <systemd-devel at lists.freedesktop.org<mailto:systemd-devel at lists.freedesktop.org>>
Subject: Re: [systemd-devel] Manual start of user@<uid>.service failed with permission denied
That sounds like it's getting immediately unmounted (or maybe not being mounted at all despite the program doing so).
Does the user-runtime-dir service continue to show as "active" after this, or does it return to "inactive"?
Does your /etc/fstab have any mentions of /run/user/1001? Or more generally, are there any run-user-1001.mount units? (If you 'systemctl status' this unit, does the status include a source path?)
On Tue, Dec 12, 2023, 17:34 Christopher Wong <Christopher.Wong at axis.com<mailto:Christopher.Wong at axis.com>> wrote:
Hi Mantas,
I currently have the following flow:
1. No /run/user/1001 directory
2. systemctl start user at 1001.service<mailto:user at 1001.service>
3. systemd start user-runtime-dir at 1001.service<mailto:user-runtime-dir at 1001.service> which ends successfully.
4. The directory /run/user/1001 exists now, but is empty, owned by root with mode 0700
5. I don’t have findmnt on my system, so I used mount, but /run/user/1001 is not listed.
6. systemd start user at 1001.service<mailto:user at 1001.service> which fails due to permission denied.
I can’t explain why the /run/user/1001 is owned by root after user-runtime-dir at 1001.service<mailto:user-runtime-dir at 1001.service> successfully exited. I added some personal print in systemd code to ensure that the mount command returned success (r=0). Although, the mount was successful the command “mount” didn’t list it. In the list of mounts starting with /run I could only find these entries:
Dec 12 16:19:35 host mount[14500]: tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
Dec 12 16:19:35 host mount[14500]: tmpfs on /run/credentials type tmpfs (ro,nosuid,nodev,noexec,mode=755)
Dec 12 16:19:35 host mount[14500]: tmpfs on /run/systemd/incoming type tmpfs (ro,nosuid,nodev,mode=755)
If I do a chown of the directory in user at 1001.service<mailto:user at 1001.service> then it works
root at host:/run/user# ls -la 1001
drwx------ 3 ida root 80 Dec 12 16:19 .
drwxr-xr-x 3 root root 60 Dec 12 16:19 ..
srw-rw-rw- 1 ida ssh-user 0 Dec 12 16:19 bus
drwxr-xr-x 5 ida ssh-user 140 Dec 12 16:19 systemd
The ”mount” command don’t list /run/user/1001 for the successful case either.
Best regards,
Christopher Wong
From: Mantas Mikulėnas <grawity at gmail.com<mailto:grawity at gmail.com>>
Date: Monday, 11 December 2023 at 17:56
To: Christopher Wong <Christopher.Wong at axis.com<mailto:Christopher.Wong at axis.com>>
Cc: Systemd <systemd-devel at lists.freedesktop.org<mailto:systemd-devel at lists.freedesktop.org>>
Subject: Re: [systemd-devel] Manual start of user@<uid>.service failed with permission denied
On Mon, Dec 11, 2023, 17:28 Christopher Wong <Christopher.Wong at axis.com<mailto:Christopher.Wong at axis.com>> wrote:
Hi Mantas,
I have added ExecStartPre to user at .service<mailto:user at .service> to run “id” and “ls -la”:
Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Will mount /run/user/1001 owned by 1001:118
Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Mounting tmpfs (tmpfs) on /run/user/1001 (MS_NOSUID|MS_NODEV "mode=0700,uid=1001,gid=118,size=99426304,nr_inodes=24274")...
Dec 11 15:50:34 host systemd[1]: Finished User Runtime Directory /run/user/1001.
Dec 11 15:50:34 host systemd[1]: Starting User Manager for UID 1001...
Dec 11 15:50:34 host id[40291]: uid=1001(ida) gid=118(ssh-users) groups=118(ssh-users),236(systemd-journal)
Dec 11 15:50:34 host ls[40293]: drwxr-xr-x 3 root root 60 Dec 11 15:50 .
Dec 11 15:50:34 host ls[40293]: drwxr-xr-x 98 root root 2120 Dec 11 15:30 ..
Dec 11 15:50:34 host ls[40293]: drwx------ 2 root root 40 Dec 11 15:50 1001
Dec 11 15:50:34 host systemd[40294]: systemd 254.7-2-g9edc143 running in user mode for user 1001/ida. (-PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT -QRENCODE -TPM2 +BZIP2 -LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON -UTMP -SYSVINIT default-hierarchy=unified)
The /run/user/1001 belongs to root with mode 0700. Should this belong to root?
No, it should be owned by UID 1001 (though still mode 0700).
Is it because I manually start user at 1001.service<mailto:user at 1001.service> as root?
Which user started the .service is usually not important, all services get a "fresh" environment that's fully described by the unit file.
So even if you did 'systemctl start' as root, the unit has User=%i so the instance parameter tells it which UID to run as, so will be running as UID 1001. Likewise user-runtime-dir at 1001 will get the UID for the mount from its instance name (you can see that the "Mounting tmpfs" message has the correct information).
However, after user-runtime-dir at 1001.service<mailto:user-runtime-dir at 1001.service> has finished it startup, the user at 1001.service<mailto:user at 1001.service> is started as uid=1001 and therefore can’t create any directories under /run/user/1001. Resulting in user at 1001.service<mailto:user at 1001.service> failed to start.
If I add “ExecStartPre=+chown %i /run/user/%i” to user at .service<mailto:user at .service> then it works! But I am unsure if this is really the way fix this.
So far, it sounds like the directory is being created *by something else* before user-runtime-dir@ is even invoked.
Try adding the same "-/bin/ls -lad /run/user/%i" as both ExecStartPre and ExecStartPost of user-runtime-dir@ (and maybe even a findmnt). If the directory already exists during ExecStartPre, start looking for other services or cronjobs, or tmpfiles.d configs, or 'su' invocations, which may cause it to be created.
There might also be something that chowns it to root *after* it was created correctly. If you actually see the tmpfs mount in 'findmnt' or in 'mount', but it's owned by root despite having uid=1001 in its mount options, something has chowned it...or your tmpfs feature is broken.
If you don't see it in findmnt at all, even after user-runtime-dir has succeeded – either the mount failed quietly, or... something (like systemd itself) has quietly unmounted it.
Regarding the testing, I have done both restart of everything and manual, but the result is the same. Now that I have the “Environment=XDG_RUNTIME_DIR=/run/user/%i” I no longer need to do “systemctl set-environment …”
Thank you for taking your time!
Best regards,
Christopher Wong
From: Mantas Mikulėnas <grawity at gmail.com<mailto:grawity at gmail.com>>
Date: Friday, 8 December 2023 at 21:53
To: Christopher Wong <Christopher.Wong at axis.com<mailto:Christopher.Wong at axis.com>>
Cc: Systemd <systemd-devel at lists.freedesktop.org<mailto:systemd-devel at lists.freedesktop.org>>
Subject: Re: [systemd-devel] Manual start of user@<uid>.service failed with permission denied
On Fri, Dec 8, 2023 at 6:53 PM Christopher Wong <Christopher.Wong at axis.com<mailto:Christopher.Wong at axis.com>> wrote:
Hi Mantas,
I have from your suggestion done the following:
Putting the below in user at .service
[Service]
...
Environment=XDG_RUNTIME_DIR=/run/user/%i
Environment=SYSTEMD_LOG_LEVEL=debug
Putting the below in user-runtime-dir at .service
[Service]
...
Environment=SYSTEMD_LOG_LEVEL=debug
Then I have disabled the global set-log-level debug (if this is also required, please let me know).
Unlike set-environment that's not global, it only affects pid1.
What I can see from the logs is that user-runtime-dir at 1001.service<mailto:user-runtime-dir at 1001.service> seems to be started and mount /run/user/1001, but addition creation of directory under this mount is getting permission denied.
Dec 08 17:33:29 host systemd[1]: Created slice User Slice of UID 1001.
Dec 08 17:33:29 host systemd[1]: Starting User Runtime Directory /run/user/1001...
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state UNSET -> OPENING
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: sd-bus: starting bus by connecting to /run/dbus/system_bus_socket...
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state OPENING -> AUTHENTICATING
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state AUTHENTICATING -> HELLO
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.2536 path=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 signature=s error-name=n/a error-message=n/a
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state HELLO -> RUNNING
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Sent message type=method_call sender=n/a destination=org.freedesktop.login1 path=/org/freedesktop/login1 interface=org.freedesktop.DBus.Properties member=Get cookie=2 reply_cookie=0 signature=ss error-name=n/a error-message=n/a
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Got message type=method_return sender=:1.323 destination=:1.2536 path=n/a interface=n/a member=n/a cookie=15 reply_cookie=2 signature=v error-name=n/a error-message=n/a
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Sent message type=method_call sender=n/a destination=org.freedesktop.login1 path=/org/freedesktop/login1 interface=org.freedesktop.DBus.Properties member=Get cookie=3 reply_cookie=0 signature=ss error-name=n/a error-message=n/a
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Got message type=method_return sender=:1.323 destination=:1.2536 path=n/a interface=n/a member=n/a cookie=16 reply_cookie=3 signature=v error-name=n/a error-message=n/a
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state RUNNING -> CLOSED
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Will mount /run/user/1001 owned by 1001:118
Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Mounting tmpfs (tmpfs) on /run/user/1001 (MS_NOSUID|MS_NODEV "mode=0700,uid=1001,gid=118,size=99426304,nr_inodes=24274")...
Dec 08 17:33:29 host systemd[1]: Finished User Runtime Directory /run/user/1001.
Dec 08 17:33:29 host systemd[1]: Starting User Manager for UID 1001...
Dec 08 17:33:29 host systemd[36280]: systemd 254.7-2-g9edc143 running in user mode for user 1001/ida. (-PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT -QRENCODE -TPM2 +BZIP2 -LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON -UTMP -SYSVINIT default-hierarchy=unified)
Dec 08 17:33:29 host systemd[36280]: Failed to create '/run/user/1001/systemd/inaccessible', ignoring: Permission denied
Dec 08 17:33:29 host systemd[36280]: Failed to create '/run/user/1001/systemd/inaccessible/reg', ignoring: Permission denied
Dec 08 17:33:29 host systemd[36280]: Failed to create '/run/user/1001/systemd/inaccessible/dir', ignoring: Permission denied
Dec 08 17:33:29 host systemd[36280]: Failed to create '/run/user/1001/systemd/inaccessible/fifo', ignoring: Permission denied
Dec 08 17:33:29 host systemd[36280]: Failed to create '/run/user/1001/systemd/inaccessible/sock', ignoring: Permission denied
Dec 08 17:33:29 host systemd[36280]: Failed to create '/run/user/1001/systemd/inaccessible/chr', ignoring: Permission denied
Dec 08 17:33:29 host systemd[36280]: Failed to create '/run/user/1001/systemd/inaccessible/blk', ignoring: Permission denied
What's the ownership of /run/user/1001 and /run/user/1001/systemd after all of this?
Are you rebooting between tests or just manually starting it?
My current guess is that due to the earlier `systemctl set-environment`, some *other* thing that's running as root inherited the /run/user/1001 path and created root-owned directories there? That's the issue with setting global environment, it needs to be unset afterwards...
--
Mantas Mikulėnas
--
Mantas Mikulėnas
--
Mantas Mikulėnas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20231213/a69f059a/attachment-0001.htm>
More information about the systemd-devel
mailing list