<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="en-SE" link="blue" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> Hi Mantas,<o:p></o:p> <o:p> </o:p> After <a href="mailto:user@1001.service">user@1001.service</a> failed, it trigger the stopping process and become inactive.<o:p></o:p> <o:p> </o:p> ○ <a href="mailto:user-runtime-dir@1001.service">user-runtime-dir@1001.service</a> - User Runtime Directory /run/user/1001<o:p></o:p> Loaded: loaded (<a href="mailto:/etc/systemd/system/user-runtime-dir@.service">/etc/systemd/system/user-runtime-dir@.service</a>; static)<o:p></o:p> Drop-In: /usr/lib/systemd/system/service.d<o:p></o:p> └─10-axis.conf, 20-axis-sandbox.conf<o:p></o:p> Active: inactive (dead) since Tue 2023-12-12 16:33:35 CET; 36min ago<o:p></o:p> Duration: 315ms<o:p></o:p> Docs: man:user@.service(5)<o:p></o:p> Process: 16325 ExecStartPre=ls -la /run/user (code=exited, status=0/SUCCESS)<o:p></o:p> Process: 16327 ExecStartPre=mount (code=exited, status=0/SUCCESS)<o:p></o:p> Process: 16329 ExecStart=/usr/lib/systemd/systemd-user-runtime-dir start 1001 (code=exited, status=0/SUCCESS)<o:p></o:p> Process: 16334 ExecStartPost=sleep 5 (code=exited, status=0/SUCCESS)<o:p></o:p> Process: 16347 ExecStartPost=ls -la /run/user/1001 (code=exited, status=0/SUCCESS)<o:p></o:p> Process: 16351 ExecStartPost=mount (code=exited, status=0/SUCCESS)<o:p></o:p> Process: 16361 ExecStop=/usr/lib/systemd/systemd-user-runtime-dir stop 1001 (code=exited, status=0/SUCCESS)<o:p></o:p> Main PID: 16329 (code=exited, status=0/SUCCESS)<o:p></o:p> CPU: 48ms<o:p></o:p> <o:p> </o:p> /etc/fstab don’t include anything on /run/user/1001 and there is no mount unit for run-user-1001.mount either.<o:p></o:p> <o:p> </o:p> <div> Best regards,<o:p></o:p> Christopher Wong<o:p></o:p> </div> <o:p> </o:p> <o:p> </o:p> <div id="mail-editor-reference-message-container"> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Mantas Mikulėnas <grawity@gmail.com> Date: Tuesday, 12 December 2023 at 17:05 To: Christopher Wong <Christopher.Wong@axis.com> Cc: Systemd <systemd-devel@lists.freedesktop.org> Subject: Re: [systemd-devel] Manual start of user@<uid>.service failed with permission denied<o:p></o:p> </div> <div> That sounds like it's getting immediately unmounted (or maybe not being mounted at all despite the program doing so).<o:p></o:p> <div> <o:p> </o:p> </div> <div> Does the user-runtime-dir service continue to show as "active" after this, or does it return to "inactive"?<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> Does your /etc/fstab have any mentions of /run/user/1001? Or more generally, are there any run-user-1001.mount units? (If you 'systemctl status' this unit, does the status include a source path?)<o:p></o:p> </div> </div> <o:p> </o:p> <div> <div> On Tue, Dec 12, 2023, 17:34 Christopher Wong <<a href="mailto:Christopher.Wong@axis.com" target="_blank">Christopher.Wong@axis.com</a>> wrote:<o:p></o:p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm"> <div> <div> Hi Mantas,<o:p></o:p> <o:p></o:p> I currently have the following flow:<o:p></o:p> <o:p></o:p> <ol start="1" type="1"> <li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1"> No /run/user/1001 directory<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1"> systemctl start <a href="mailto:user@1001.service" target="_blank"> user@1001.service</a><o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1"> systemd start <a href="mailto:user-runtime-dir@1001.service" target="_blank"> user-runtime-dir@1001.service</a> which ends successfully.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1"> The directory /run/user/1001 exists now, but is empty, owned by root with mode 0700<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1"> I don’t have findmnt on my system, so I used mount, but /run/user/1001 is not listed.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1"> systemd start <a href="mailto:user@1001.service" target="_blank"> user@1001.service</a> which fails due to permission denied.<o:p></o:p></li></ol> <o:p></o:p> I can’t explain why the /run/user/1001 is owned by root after <a href="mailto:user-runtime-dir@1001.service" target="_blank">user-runtime-dir@1001.service</a> successfully exited. I added some personal print in systemd code to ensure that the mount command returned success (r=0). Although, the mount was successful the command “mount” didn’t list it. In the list of mounts starting with /run I could only find these entries:<o:p></o:p> <o:p></o:p> Dec 12 16:19:35 host mount[14500]: tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)<o:p></o:p> Dec 12 16:19:35 host mount[14500]: tmpfs on /run/credentials type tmpfs (ro,nosuid,nodev,noexec,mode=755)<o:p></o:p> Dec 12 16:19:35 host mount[14500]: tmpfs on /run/systemd/incoming type tmpfs (ro,nosuid,nodev,mode=755)<o:p></o:p> <o:p></o:p> If I do a chown of the directory in <a href="mailto:user@1001.service" target="_blank">user@1001.service</a> then it works<o:p></o:p> <o:p></o:p> root@host:/run/user# ls -la 1001<o:p></o:p> drwx------ 3 ida root 80 Dec 12 16:19 .<o:p></o:p> drwxr-xr-x 3 root root 60 Dec 12 16:19 ..<o:p></o:p> srw-rw-rw- 1 ida ssh-user 0 Dec 12 16:19 bus<o:p></o:p> drwxr-xr-x 5 ida ssh-user 140 Dec 12 16:19 systemd<o:p></o:p> <o:p></o:p> The ”mount” command don’t list /run/user/1001 for the successful case either.<o:p></o:p> <o:p></o:p> <div> Best regards,<o:p></o:p> Christopher Wong<o:p></o:p> </div> <o:p></o:p> <o:p></o:p> <div id="m_-8990293804925907923m_50470161548180358mail-editor-reference-message-container"> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Mantas Mikulėnas <<a href="mailto:grawity@gmail.com" target="_blank">grawity@gmail.com</a>> Date: Monday, 11 December 2023 at 17:56 To: Christopher Wong <<a href="mailto:Christopher.Wong@axis.com" target="_blank">Christopher.Wong@axis.com</a>> Cc: Systemd <<a href="mailto:systemd-devel@lists.freedesktop.org" target="_blank">systemd-devel@lists.freedesktop.org</a>> Subject: Re: [systemd-devel] Manual start of user@<uid>.service failed with permission denied<o:p></o:p> </div> <div> <div> <div> <div> On Mon, Dec 11, 2023, 17:28 Christopher Wong <<a href="mailto:Christopher.Wong@axis.com" target="_blank">Christopher.Wong@axis.com</a>> wrote:<o:p></o:p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt"> <div> <div> Hi Mantas,<o:p></o:p> <o:p></o:p> I have added ExecStartPre to <a href="mailto:user@.service" target="_blank">user@.service</a> to run “id” and “ls -la”:<o:p></o:p> <o:p></o:p> Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Will mount /run/user/1001 owned by 1001:118<o:p></o:p> Dec 11 15:50:34 host systemd-user-runtime-dir[40287]: Mounting tmpfs (tmpfs) on /run/user/1001 (MS_NOSUID|MS_NODEV "mode=0700,uid=1001,gid=118,size=99426304,nr_inodes=24274")...<o:p></o:p> Dec 11 15:50:34 host systemd[1]: Finished User Runtime Directory /run/user/1001.<o:p></o:p> Dec 11 15:50:34 host systemd[1]: Starting User Manager for UID 1001...<o:p></o:p> Dec 11 15:50:34 host id[40291]: uid=1001(ida) gid=118(ssh-users) groups=118(ssh-users),236(systemd-journal)<o:p></o:p> Dec 11 15:50:34 host ls[40293]: drwxr-xr-x 3 root root 60 Dec 11 15:50 .<o:p></o:p> Dec 11 15:50:34 host ls[40293]: drwxr-xr-x 98 root root 2120 Dec 11 15:30 ..<o:p></o:p> Dec 11 15:50:34 host ls[40293]: drwx------ 2 root root 40 Dec 11 15:50 1001<o:p></o:p> Dec 11 15:50:34 host systemd[40294]: systemd 254.7-2-g9edc143 running in user mode for user 1001/ida. (-PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT -QRENCODE -TPM2 +BZIP2 -LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON -UTMP -SYSVINIT default-hierarchy=unified)<o:p></o:p> <o:p></o:p> The /run/user/1001 belongs to root with mode 0700. Should this belong to root?<o:p></o:p> </div> </div> </blockquote> </div> </div> <div> No, it should be owned by UID 1001 (though still mode 0700).<o:p></o:p> </div> <div> <div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt"> <div> <div> Is it because I manually start <a href="mailto:user@1001.service" target="_blank">user@1001.service</a> as root?<o:p></o:p> </div> </div> </blockquote> </div> </div> <div> Which user started the .service is usually not important, all services get a "fresh" environment that's fully described by the unit file.<o:p></o:p> </div> <div> <o:p></o:p> </div> <div> So even if you did 'systemctl start' as root, the unit has User=%i so the instance parameter tells it which UID to run as, so will be running as UID 1001. Likewise user-runtime-dir@1001 will get the UID for the mount from its instance name (you can see that the "Mounting tmpfs" message has the correct information).<o:p></o:p> </div> <div> <div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt"> <div> <div> However, after <a href="mailto:user-runtime-dir@1001.service" target="_blank">user-runtime-dir@1001.service</a> has finished it startup, the <a href="mailto:user@1001.service" target="_blank">user@1001.service</a> is started as uid=1001 and therefore can’t create any directories under /run/user/1001. Resulting in <a href="mailto:user@1001.service" target="_blank">user@1001.service</a> failed to start.<o:p></o:p> <o:p></o:p> If I add “ExecStartPre=+chown %i /run/user/%i” to <a href="mailto:user@.service" target="_blank">user@.service</a> then it works! But I am unsure if this is really the way fix this.<o:p></o:p> </div> </div> </blockquote> </div> </div> <div> <o:p></o:p> </div> <div> So far, it sounds like the directory is being created *by something else* before user-runtime-dir@ is even invoked.<o:p></o:p> </div> <div> <o:p></o:p> </div> <div> Try adding the same "-/bin/ls -lad /run/user/%i" as both ExecStartPre and ExecStartPost of user-runtime-dir@ (and maybe even a findmnt). If the directory already exists during ExecStartPre, start looking for other services or cronjobs, or tmpfiles.d configs, or 'su' invocations, which may cause it to be created.<o:p></o:p> </div> <div> <o:p></o:p> </div> <div> There might also be something that chowns it to root *after* it was created correctly. If you actually see the tmpfs mount in 'findmnt' or in 'mount', but it's owned by root despite having uid=1001 in its mount options, something has chowned it...or your tmpfs feature is broken.<o:p></o:p> </div> <div> <o:p></o:p> </div> <div> If you don't see it in findmnt at all, even after user-runtime-dir has succeeded – either the mount failed quietly, or... something (like systemd itself) has quietly unmounted it.<o:p></o:p> </div> <div> <o:p></o:p> </div> <div> <div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt"> <div> <div> <o:p></o:p> Regarding the testing, I have done both restart of everything and manual, but the result is the same. Now that I have the “Environment=XDG_RUNTIME_DIR=/run/user/%i” I no longer need to do “systemctl set-environment …”<o:p></o:p> <o:p></o:p> Thank you for taking your time!<o:p></o:p> <o:p></o:p> <div> Best regards,<o:p></o:p> Christopher Wong<o:p></o:p> </div> <o:p></o:p> <o:p></o:p> <div id="m_-8990293804925907923m_50470161548180358m_-2724165164821707795mail-editor-reference-message-container"> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Mantas Mikulėnas <<a href="mailto:grawity@gmail.com" target="_blank">grawity@gmail.com</a>> Date: Friday, 8 December 2023 at 21:53 To: Christopher Wong <<a href="mailto:Christopher.Wong@axis.com" target="_blank">Christopher.Wong@axis.com</a>> Cc: Systemd <<a href="mailto:systemd-devel@lists.freedesktop.org" target="_blank">systemd-devel@lists.freedesktop.org</a>> Subject: Re: [systemd-devel] Manual start of user@<uid>.service failed with permission denied<o:p></o:p> </div> <div> <div> <div> On Fri, Dec 8, 2023 at 6:53 PM Christopher Wong <<a href="mailto:Christopher.Wong@axis.com" target="_blank">Christopher.Wong@axis.com</a>> wrote:<o:p></o:p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt"> <div> <div> <div> Hi Mantas,<o:p></o:p> <o:p></o:p> I have from your suggestion done the following:<o:p></o:p> <o:p></o:p> Putting the below in user@.service<o:p></o:p> <o:p></o:p> [Service]<o:p></o:p> ...<o:p></o:p> Environment=XDG_RUNTIME_DIR=/run/user/%i<o:p></o:p> Environment=SYSTEMD_LOG_LEVEL=debug<o:p></o:p> <o:p></o:p> Putting the below in user-runtime-dir@.service<o:p></o:p> <o:p></o:p> [Service]<o:p></o:p> ...<o:p></o:p> Environment=SYSTEMD_LOG_LEVEL=debug<o:p></o:p> <o:p></o:p> Then I have disabled the global set-log-level debug (if this is also required, please let me know).<o:p></o:p> </div> </div> </div> </blockquote> <div> <o:p></o:p> </div> <div> Unlike set-environment that's not global, it only affects pid1.<o:p></o:p> </div> <div> <o:p></o:p> </div> <div> <o:p></o:p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt"> <div> <div> <div> What I can see from the logs is that <a href="mailto:user-runtime-dir@1001.service" target="_blank">user-runtime-dir@1001.service</a> seems to be started and mount /run/user/1001, but addition creation of directory under this mount is getting permission denied. <o:p></o:p> <o:p></o:p> Dec 08 17:33:29 host systemd[1]: Created slice User Slice of UID 1001.<o:p></o:p> Dec 08 17:33:29 host systemd[1]: Starting User Runtime Directory /run/user/1001...<o:p></o:p> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state UNSET -> OPENING<o:p></o:p> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: sd-bus: starting bus by connecting to /run/dbus/system_bus_socket...<o:p></o:p> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state OPENING -> AUTHENTICATING<o:p></o:p> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state AUTHENTICATING -> HELLO<o:p></o:p> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a<o:p></o:p> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.2536 path=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 signature=s error-name=n/a error-message=n/a<o:p></o:p> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state HELLO -> RUNNING<o:p></o:p> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Sent message type=method_call sender=n/a destination=org.freedesktop.login1 path=/org/freedesktop/login1 interface=org.freedesktop.DBus.Properties member=Get cookie=2 reply_cookie=0 signature=ss error-name=n/a error-message=n/a<o:p></o:p> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Got message type=method_return sender=:1.323 destination=:1.2536 path=n/a interface=n/a member=n/a cookie=15 reply_cookie=2 signature=v error-name=n/a error-message=n/a<o:p></o:p> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Sent message type=method_call sender=n/a destination=org.freedesktop.login1 path=/org/freedesktop/login1 interface=org.freedesktop.DBus.Properties member=Get cookie=3 reply_cookie=0 signature=ss error-name=n/a error-message=n/a<o:p></o:p> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Got message type=method_return sender=:1.323 destination=:1.2536 path=n/a interface=n/a member=n/a cookie=16 reply_cookie=3 signature=v error-name=n/a error-message=n/a<o:p></o:p> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Bus n/a: changing state RUNNING -> CLOSED<o:p></o:p> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Will mount /run/user/1001 owned by 1001:118<o:p></o:p> Dec 08 17:33:29 host systemd-user-runtime-dir[36278]: Mounting tmpfs (tmpfs) on /run/user/1001 (MS_NOSUID|MS_NODEV "mode=0700,uid=1001,gid=118,size=99426304,nr_inodes=24274")...<o:p></o:p> Dec 08 17:33:29 host systemd[1]: Finished User Runtime Directory /run/user/1001.<o:p></o:p> Dec 08 17:33:29 host systemd[1]: Starting User Manager for UID 1001...<o:p></o:p> Dec 08 17:33:29 host systemd[36280]: systemd 254.7-2-g9edc143 running in user mode for user 1001/ida. (-PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT -QRENCODE -TPM2 +BZIP2 -LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON -UTMP -SYSVINIT default-hierarchy=unified)<o:p></o:p> Dec 08 17:33:29 host systemd[36280]: Failed to create '/run/user/1001/systemd/inaccessible', ignoring: Permission denied<o:p></o:p> Dec 08 17:33:29 host systemd[36280]: Failed to create '/run/user/1001/systemd/inaccessible/reg', ignoring: Permission denied<o:p></o:p> Dec 08 17:33:29 host systemd[36280]: Failed to create '/run/user/1001/systemd/inaccessible/dir', ignoring: Permission denied<o:p></o:p> Dec 08 17:33:29 host systemd[36280]: Failed to create '/run/user/1001/systemd/inaccessible/fifo', ignoring: Permission denied<o:p></o:p> Dec 08 17:33:29 host systemd[36280]: Failed to create '/run/user/1001/systemd/inaccessible/sock', ignoring: Permission denied<o:p></o:p> Dec 08 17:33:29 host systemd[36280]: Failed to create '/run/user/1001/systemd/inaccessible/chr', ignoring: Permission denied<o:p></o:p> Dec 08 17:33:29 host systemd[36280]: Failed to create '/run/user/1001/systemd/inaccessible/blk', ignoring: Permission denied<o:p></o:p> </div> </div> </div> </blockquote> <div> <o:p></o:p> </div> <div> What's the ownership of /run/user/1001 and /run/user/1001/systemd after all of this?<o:p></o:p> </div> <div> <o:p></o:p> </div> <div> Are you rebooting between tests or just manually starting it?<o:p></o:p> </div> <div> <o:p></o:p> </div> <div> My current guess is that due to the earlier `systemctl set-environment`, some *other* thing that's running as root inherited the /run/user/1001 path and created root-owned directories there? That's the issue with setting global environment, it needs to be unset afterwards...<o:p></o:p> </div> </div> -- <o:p></o:p> <div> <div> Mantas Mikulėnas<o:p></o:p> </div> </div> </div> </div> </div> </div> </div> </blockquote> </div> </div> </div> </div> </div> </div> </div> </blockquote> </div> </div> </div> </div> </body> </html>