[systemd-devel] Concurrent login / daemon-reload produces abandoned sessions

[Nicolas Bock]

On Thu, Jul 08 2021, Michal Koutný wrote:

> Hello Nicolas.
>
> On Wed, Jul 07, 2021 at 02:12:51PM +0200, Nicolas Bock <nicolas.bock at canonical.com> wrote:
>> Using systemd-248.3-1ubuntu1 on Ubuntu Impish the following
>> script produces multiple abandoned sessions:
>> 
>> 	$ for i in {1..100}; do sleep 0.2; ssh localhost sudo systemctl daemon-reload & ssh localhost sleep 1 & done
>> 	$ sleep 2
>> 	$ jobs -p | xargs --verbose --no-run-if-empty kill -KILL
>> 	$ systemctl | grep abandoned
>> 	  session-174.scope                    loaded active     abandoned Session 174 of user ubuntu
>> 	  session-175.scope                    loaded active     abandoned Session 175 of user ubuntu
>> 	  session-176.scope                    loaded active     abandoned Session 176 of user ubuntu
>> 	  session-25.scope                     loaded active     abandoned Session 25 of user ubuntu
>> 
>> I would like to debug this behavior further and understand
>> why this is happening but don't know where to look next.
>
> It might be a bit challenging :)
>
>> Is there any information in particular I should look at?
>
> I assume you use hybrid or unified cgroup setup and that the abandoned
> scopes are empty (no processes in their cgroups), correct?

Yes, you are correct. Ubuntu builds their systemd with
`-Ddefault-hierarchy=hybrid`. And yes, the abandoned scopes
have no processes.

> My hypothesis is following
>
> // race between scope abandonement, emptiness notification -> abandon comes first
> manager_reload
>   manager_clear_jobs_and_units
>     unit_release_cgroup
>       inotify_rm_watch(u->manager->cgroup_inotify_fd, u->cgroup_control_inotify_wd)
> [...]
> // last process terminates somewhere here but we're not watching emptiness yet
> scope_coldplug()
>   // scope should be checked for emptiness here
>
> I _think_ this could be fixed with the patch
> --- a/src/core/scope.c
> +++ b/src/core/scope.c
> @@ -243,8 +243,8 @@ static int scope_coldplug(Unit *u) {
>                                  if (r < 0 && r != -EEXIST)
>                                          return r;
>                          }
> -                } else
> -                        (void) unit_enqueue_rewatch_pids(u);
> +                }
> +                (void) unit_enqueue_rewatch_pids(u);
>          }
>
>          bus_scope_track_controller(s);
>
> Can you file a Github issue to track this (and possibly try if this
> works for you)?

I have already opened an issue [1] but it was closed. Maybe
we can just re-open it?

I patched the impish systemd package but the issue is still
present.

Thanks,

Nick

[1] https://github.com/systemd/systemd/issues/20136