[systemd-devel] Debugging systemd in a container

[Lennart Poettering]

On So, 10.05.20 20:57, Mark Clarkson (mark at smorg.co.uk) wrote:

> Hi,
> I finally got:
>
> Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not
> permitted
> [!!!!!!] Failed to mount API filesystems, freezing.
>
> But I still can't get debug logs - any ideas?
>
> Anyway, I followed through the systemd source and it seems that it's
> trying to mount cgroupv1 rather than cgroupv2. That error should have
> said: 'Failed to mount cgroup2 at...' but it says cgroup.
>
> My host system /proc/cmdline is:
>
> BOOT_IMAGE=(hd0,gpt5)/vmlinuz-5.6.10-300.fc32.x86_64
> root=/dev/mapper/fedora_ux305-root ro rd.lvm.lv=fedora_ux305/swap
> rd.lvm.lv=fedora_ux305/root rhgb quiet
> systemd.unified_cgroup_hierarchy=1
>
> And I set:
>
> ENTRYPOINT ["/usr/local/bin/entrypoint", "/sbin/init",
> "systemd.unified_cgroup_hierarchy=1", "cgroup_no_v1=all", "debug"]
>
> The `/usr/local/bin/entrypoint` shell script sets things up before
> doing `exec "$@"`. No matter what I do, changing cmdline options
> (probably incorrectly!), I can't get systemd to stop trying to mount
> cgroupv1.
>
> What I think I want:
>
> * bind mount podman's cgroupv2 at /sys/fs/cgroup. For example
>   mount --bind /sys/fs/cgroup/machine.slice/libpod-xxx /sys/fs/cgroup
> * systemd starts and starts dbus etc.
> * cri-o/kubelet can then do whatever they want in /sys/fs/cgroup and
>   won't get confused.
>
> Is that possible? I did a different set of bind mounts for cgroupv1
> previously so crio couldn't see other cgroups and systemd was happy
> (/proc/PID/cgroup was still correct) and crio and kubelet were also
> happy (not able to see other cgroup containers).

This is probably more a question you have to ask the podman people
than the systemd people.

systemd's expectations regarding the container manager are documented
here:

https://systemd.io/CONTAINER_INTERFACE

The systemd.unified_cgroup_hierarchy=1 switch is only honoured if
cgroupfs is not yet mounted in the container, in which case systemd
will try to mount it. Generally, it's a good idea to pre-mount it
though, as you typically want the same cgroups logic inside and
outside the container. systemd tries to adapt to the environment it
runs in, and that means cgroupsv2 is used when it is already mounted
like that, and cgroupsv1 otherwise.

Lennart

--
Lennart Poettering, Berlin