<HTML><BODY><div><div>Thank you for checking!<br><br>Yes, it clearly seems that systemd and kubelet in such setup shares cgroups which is not supposed.<br>We will prioritize moving our cluster to use systemd cgroup driver to avoid such conflict.<br>Also I think it would be good to have extra check on kubelet side to avoid running cgroupfs driver on systemd systems. But it’s question to k8s folks which already rised in slack.</div><div> </div><div>-----</div><div>Just out of curiosity, how systemd in particular may be disrupted with such record in root of it’s cgroups hierarchy as /kubpods/bla/bla during service (de)activation?<br>Or how it may disrupt the kubelet or workload running by it?<br><br>Will it delete such records because of some logic? Or there will be name conflict during cgroup creation?<br>Would be happy to know more details of cgroups interference.<br><br>I've read few articles:<br>https://systemd.io/CGROUP_DELEGATION/<br>http://0pointer.de/blog/projects/cgroups-vs-cgroups.html<br>https://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/<br>https://www.freedesktop.org/wiki/Software/systemd/writing-vm-managers/<br><br>even outdated one<br>https://www.freedesktop.org/wiki/Software/systemd/PaxControlGroups/<br><br>Seems I missed some technical details how exact it will interfere.<br>-----</div><div><br>> It may be a residual inside kubelet context when environment was prepared for a container spawned from within this context</div><div><br>Just last finding of this weird cgroup mount:<pre><code># find / -name '*8842def24*'
/sys/fs/cgroup/systemd/kubepods/burstable/pod7ffde41a-fa85-4b01-8023-69a4e4b50c55/8842def241fac72cb34fdce90297b632f098289270fa92ec04643837f5748c15
/sys/fs/cgroup/systemd/machine.slice/systemd-nspawn@centos75.service/payload/system.slice/host\x2drootfs-sys-fs-cgroup-systemd-kubepods-burstable-pod7ffde41a\x2dfa85\x2d4b01\x2d8023\x2d69a4e4b50c55-8842def241fac72cb34fdce90297b632f098289270fa92ec04643837f5748c15.mount
and
</code>
</pre><pre><code># machinectl list
MACHINE CLASS SERVICE OS VERSION ADDRESSES
centos75 container systemd-nspawn centos 7 -
frr container systemd-nspawn ubuntu 18.04 -
2 machines listed.</code></pre>Since container with id <code>8842def241 </code>is not running it’s hard to understand what exactly happened, who did such mount and reproduce the conflict situation.<br><br>May I ask, how systemd-nspawn may be involved in it? Or any ideas what happened so I still have two times mounted the systemd named hierarchy?<br> </div><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;">Thursday, November 19, 2020 3:25 AM +09:00 from Michal Koutný <mkoutny@suse.com>:<br> <div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_16057239522114890934_BODY">Thanks for the details.<br><br>On Mon, Nov 16, 2020 at 09:30:20PM +0300, Andrei Enshin <<a href="/compose?To=b1os@bk.ru">b1os@bk.ru</a>> wrote:<br>> I see the kubelet crash with error: «Failed to start ContainerManager failed to initialize top level QOS containers: root container [kubepods] doesn't exist»<br>> details: <a href="https://github.com/kubernetes/kubernetes/issues/95488" target="_blank">https://github.com/kubernetes/kubernetes/issues/95488</a><br>I skimmed the issue and noticed that your setup uses 'cgroupfs' cgroup<br>driver. As explained in the other messages in this thread, it conflicts<br>with systemd operation over the root cgroup tree.<br><br>> I can see same two mounts of named systemd hierarchy from shell on the same node, simply by `$ cat /proc/self/mountinfo`<br>> I think kubelet is running in the «main» mount namespace which has weird named systemd mount.<br>I assume so as well. It may be a residual inside kubelet context when<br>environment was prepared for a container spawned from within this<br>context.<br><br>> I would like to reproduce such weird mount to understand the full<br>> situation and make sure I can avoid it in future.<br>I'm afraid you may be seeing results of various races between systemd<br>service (de)activation and container spawnings under the "shared" root<br>(both of which comprise cgroup creation/removal and migrations).<br>There's a reason behind the cgroup subtree delegation.<br><br>So I'd say there's not much to do from systemd side now.<br><br><br>Michal<br> </div></div></div></div></blockquote> <div> </div><div data-signature-widget="container"><div data-signature-widget="content"><p>---</p><p>Best Regards,<br>Andrei Enshin</p></div></div><div> </div></div></BODY></HTML>