[systemd-devel] Debugging systemd in a container

[Mark Clarkson]

On Sun, 2020-05-10 at 01:28 +0100, Mark Clarkson wrote:
> 
> I would really like to get some information out of systemd to work
> through the errors, but I can't work out how to do it. I expected it
> to
> output to stdout and be accessible through 'podman logs <ID>'.
> 
> I need logs!! :)
> 
> Thanks!
> Mark
> 

Hi,
I finally got:

Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not
permitted
[!!!!!!] Failed to mount API filesystems, freezing.

But I still can't get debug logs - any ideas?

Anyway, I followed through the systemd source and it seems that it's
trying to mount cgroupv1 rather than cgroupv2. That error should have
said: 'Failed to mount cgroup2 at...' but it says cgroup.

My host system /proc/cmdline is:

BOOT_IMAGE=(hd0,gpt5)/vmlinuz-5.6.10-300.fc32.x86_64
root=/dev/mapper/fedora_ux305-root ro rd.lvm.lv=fedora_ux305/swap
rd.lvm.lv=fedora_ux305/root rhgb quiet
systemd.unified_cgroup_hierarchy=1

And I set:

ENTRYPOINT ["/usr/local/bin/entrypoint", "/sbin/init",
"systemd.unified_cgroup_hierarchy=1", "cgroup_no_v1=all", "debug"]

The `/usr/local/bin/entrypoint` shell script sets things up before
doing `exec "$@"`. No matter what I do, changing cmdline options
(probably incorrectly!), I can't get systemd to stop trying to mount
cgroupv1.

What I think I want:

* bind mount podman's cgroupv2 at /sys/fs/cgroup. For example
  mount --bind /sys/fs/cgroup/machine.slice/libpod-xxx /sys/fs/cgroup
* systemd starts and starts dbus etc.
* cri-o/kubelet can then do whatever they want in /sys/fs/cgroup and
  won't get confused.

Is that possible? I did a different set of bind mounts for cgroupv1
previously so crio couldn't see other cgroups and systemd was happy
(/proc/PID/cgroup was still correct) and crio and kubelet were also
happy (not able to see other cgroup containers).