[systemd-devel] Permission/updating problems; different behaviour of two identical nspawn containers
Olaf the Lost Viking
olaf.the.lost.viking at gmail.com
Wed Aug 30 15:24:56 UTC 2017
Hi ML,
currently I am seeing differences between two, what I consider identical,
nspawn-containers which prevents me to update one of them. (Lots of) details
are at the end of the mail.
I set up two (hopefully) identical debian containers in nspawn for a single
service (DNS) on a debian host. Today's "apt upgrade" now throws permissions
problem on _one_ of the containers (ns4 fails, all others still work - ns3
should be identical but some service data):
root at ns4:~# apt upgrade
...
75 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 50.0 MB of archives.
After this operation, 313 kB of additional disk space will be used.
W: chown to _apt:root of directory /var/cache/apt/archives/partial failed -
_ SetupAPTPartialDirectory (1: Operation not permitted)
Do you want to continue? [Y/n]
Downloading works, but then moving the archives fails:
...
E: Failed to fetch http://security.debian.org/pool/updates/main/p/
_ postgresql-9.6/postgresql-9.6_9.6.4-0+deb9u1_amd64.deb rename failed,
_ Permission denied (/var/cache/apt/archives/partial/
_ postgresql-9.6_9.6.4-0+deb9u1_amd64.deb -> /var/cache/apt/archives/
_ postgresql-9.6_9.6.4-0+deb9u1_amd64.deb).
E: Unable to fetch some archives, maybe run apt-get update or try with --
_ fix-missing?
root at ns4:~#
I also cannot set the correct container group on the host! (Please see an
example at the very end of the mail.) Neither in the HOST, nor in the ns4
journal anything is shown.
Following I try to give as much information I consider as relevant as I can.
Please do not hesitate to ask for more details. The system is not critical and
can be rebooted (which I already did) or whatever.
Thanks a lot!
== Host
root at HOST:~# cat /etc/debian_version
9.1
root at HOST:~# systemd --v
systemd 232
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP
_ +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN
root at HOST:~# machinectl list
MACHINE CLASS SERVICE OS VERSION ADDRESSES
ns3 container systemd-nspawn debian 9 10.225.32.1...
ns4 container systemd-nspawn debian 9 10.225.64.1...
nsrec2 container systemd-nspawn debian 9 10.225.1.1...
3 machines listed.
root at HOST:~#
== nspawn container 1 (ns3) ==
root at ns3:~# cat /etc/debian_version
9.1
root at ns3:~# systemd --v
systemd 232
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP
_ +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN
== nspawn container 2 (ns4) ==
root at ns4:~# cat /etc/debian_version
9.1
root at ns4:~# systemd --v
systemd 232
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP
_ +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN
The configuration of both containers look the same to me:
== nspawn config
root at HOST:~# cat /etc/systemd/nspawn/ns3.nspawn
[Exec]
# -> guid parse bug in the kernel
#PrivateUsers=false
[Files]
# -> dynamic uid mounts apt w/o root access
#Bind=/var/cache/apt/
#Bind=/var/lib/apt/
root at HOST:~# diff /etc/systemd/nspawn/ns3.nspawn /etc/systemd/nspawn/
ns4.nspawn
root at HOST:~#
== mount config
root at HOST:~# cat /etc/systemd/system/var-lib-machines-ns3.mount
[Unit]
Before=local-fs.target
[Install]
WantedBy=local-fs.target
[Mount]
What=/dev/disk/by-label/virt
Where=/var/lib/machines/ns3/
Type=btrfs
Options=noatime,nodiratime,subvol=vm-ns3_rootfs at active
root at HOST:~# cat /etc/systemd/system/var-lib-machines-ns3-var-cache.mount
[Unit]
Before=local-fs.target
[Install]
WantedBy=local-fs.target
[Mount]
What=/dev/disk/by-label/virt
Where=/var/lib/machines/ns3//var/cache
Type=btrfs
Options=noatime,nodiratime,nodev,nosuid,noexec,subvol=vm-ns3_var-
_ cache at active
root at HOST:~#
root at HOST:~# diff /etc/systemd/system/var-lib-machines-ns3.mount /etc/
_ systemd/system/var-lib-machines-ns4.mount
9c9
< Where=/var/lib/machines/ns3/
---
> Where=/var/lib/machines/ns4/
11c11
< Options=noatime,nodiratime,subvol=vm-ns3_rootfs at active
---
> Options=noatime,nodiratime,subvol=vm-ns4_rootfs at active
root at HOST:~# diff /etc/systemd/system/var-lib-machines-ns3-var-cache.mount /
_ etc/systemd/system/var-lib-machines-ns4-var-cache.mount
9c9
< Where=/var/lib/machines/ns3//var/cache
---
> Where=/var/lib/machines/ns4//var/cache
11c11
< Options=noatime,nodiratime,nodev,nosuid,noexec,subvol=vm-ns3_var-
_ cache at active
---
> Options=noatime,nodiratime,nodev,nosuid,noexec,subvol=vm-ns4_var-
_ cache at active
root at HOST:~#
root at HOST:~# mount | grep 'ns[34].*/cache'
/dev/mapper/volg-virt on /var/lib/machines/ns4/var/cache type btrfs
_ (rw,nosuid,nodev,noexec,noatime,nodiratime,space_cache,subvolid=331,subvol=/
_ vm-ns4_var-cache at active)
/dev/mapper/volg-virt on /var/lib/machines/ns3/var/cache type btrfs
_ (rw,nosuid,nodev,noexec,noatime,nodiratime,space_cache,subvolid=350,subvol=/
_ vm-ns3_var-cache at active)
root at HOST:~#
root at HOST:~# btrfs subvolume list /var/lib/btrfs/ | grep
_ 'ns[34].*cache'
ID 331 gen 68872 top level 5 path vm-ns4_var-cache at active
ID 350 gen 67791 top level 5 path vm-ns3_var-cache at active
root at HOST:~#
== file permissions
root at HOST:~# ls -l /var/lib/machines/ns3/
total 4
...
drwxr-xr-x 1 vu-ns3-0 vg-ns3-0 100 Apr 26 12:33 var
root at HOST:~# ls -l /var/lib/machines/ns3/var/
total 8
...
drwxr-xr-x 1 vu-ns3-0 vg-ns3-0 120 May 1 20:48 cache
...
root at HOST:~# ls -l /var/lib/machines/ns3/var/cache/
total 4
drwxr-xr-x 1 vu-ns3-0 vg-ns3-0 70 May 1 20:47 apt
...
root at HOST:~# ls -l /var/lib/machines/ns3/var/cache/apt/
total 1044
drwxr-xr-x 1 vu-ns3-0 vg-ns3-0 22 Aug 30 14:48 archives
-rw-r--r-- 1 vu-ns3-0 vg-ns3-0 641725 Apr 29 12:31 pkgcache.bin
-rw-r--r-- 1 vu-ns3-0 vg-ns3-0 425316 Apr 29 12:31 srcpkgcache.bin
root at HOST:~# ls -l /var/lib/machines/ns3/var/cache/apt/archives/
total 0
-rw-r----- 1 vu-ns3-0 vg-ns3-0 0 May 1 20:47 lock
drwx------ 1 vu-ns3-104 vg-ns3-0 0 Aug 30 14:41 partial
root at HOST:~#
root at HOST:~# ls -l /var/lib/machines/ns4
total 4
...
drwxr-xr-x 1 vu-ns4-0 vg-ns4-0 100 Apr 26 12:33 var
root at HOST:~# ls -l /var/lib/machines/ns4/var/
total 8
...
drwxr-xr-x 1 vu-ns4-0 vg-ns4-0 120 Apr 28 22:07 cache
...
root at HOST:~# ls -l /var/lib/machines/ns4/var/cache/
total 4
drwxr-xr-x 1 vu-ns4-0 vg-ns4-0 70 Apr 29 12:31 apt
...
root at HOST:~# ls -l /var/lib/machines/ns4/var/cache/apt/
total 51920
drwxr-xr-x 1 vu-ns4-0 root 22 Aug 30 14:49 archives
-rw-r--r-- 1 vu-ns4-0 root 26581616 Apr 29 12:31 pkgcache.bin
-rw-r--r-- 1 vu-ns4-0 root 26581534 Apr 29 12:31 srcpkgcache.bin
root at HOST:~# ls -l /var/lib/machines/ns4/var/cache/apt/archives/
total 0
-rw-r----- 1 vu-ns4-0 vg-ns4-0 0 Apr 28 22:04 lock
drwx------ 1 vu-ns4-104 root 5000 Aug 30 17:01 partial
root at HOST:~#
== Problems
As you could see the few lines above, the groups in ns4 aren't correct for
certain files/directories. But correcting them in the guest as well as the
host fails:
root at ns4:/var/cache/apt/archives# ls -l
total 0
-rw-r----- 1 root root 0 Apr 28 22:04 lock
drwx------ 1 _apt nogroup 5000 Aug 30 17:01 partial
root at ns4:/var/cache/apt/archives# chgrp root partial/
chgrp: changing group of 'partial/': Operation not permitted
root at ns4:/var/cache/apt/archives#
root at HOST:/var/lib/machines/ns4/var/cache/apt/archives# ls -l
total 0
-rw-r----- 1 vu-ns4-0 vg-ns4-0 0 Apr 28 22:04 lock
drwx------ 1 vu-ns4-104 root 5000 Aug 30 17:01 partial
root at HOST:/var/lib/machines/ns4/var/cache/apt/archives# chgrp vg-ns4-0
_ partial/
root at HOST:/var/lib/machines/ns4/var/cache/apt/archives# ls -l
total 0
-rw-r----- 1 vu-ns4-0 vg-ns4-0 0 Apr 28 22:04 lock
drwx------ 1 vu-ns4-104 root 5000 Aug 30 17:01 partial
root at HOST:/var/lib/machines/ns4/var/cache/apt/archives#
More information about the systemd-devel
mailing list