[systemd-devel] ExecStop upon un-expected but clean shutdown

[Mathieu Lacage]

Hi,

While trying to make progress on this issue, I found this:
https://lists.freedesktop.org/archives/systemd-devel/2015-February/028058.html
which appears to be a similar problem. I eventually identified changeset
d54ddab8cbad46290306fc6e3346089fe3772d5c as the relevant change by Lennart.
A quick check in my server's distribution source package shows that this
change is already included in my server's systemd yet, I can still observe
the infinite loop described in my first email when someone kills my service
without systemctl: ExecStop is executed and blocks forever because it
attempts to connect to the service's activation socket which is under the
control of systemd itself.

Am I supposed to re-implement my ExecStop command to not use the activation
socket ?

Mathieu

On Mon, Mar 21, 2016 at 10:44 AM, Mathieu Lacage <mathieu.lacage at gmail.com>
wrote:

> Hi,
>
> I have a daemon that is socket-activated through a systemd-managed
> localhost TCP socket. The daemon is normally shutdown by a simple systemctl
> stop with ExecStop=curl -X POST on the right HTTP endpoint which makes it
> easy to cleanup resources and make sure my data hits the hard disk.
>
> Now, it just so happens that sometimes my users and myself screw up and we
> actually invoke the stop HTTP endpoint by hand because we are dumb. When
> this happens, systemd detects that the daemon dies peacefully (status = 0)
> and then proceeds to invoke ExecStop=curl -X POST (service_enter_running
> calls service_enter_stop). The latter never completes because systemd still
> has the activation socket open so curl waits forever for the TCP connection
> to be accepted (modulo timeout). When the timeout expires, systemd tries to
> re-run ExecStop. When I "kill -9" the command by hand, systemd tries to
> re-run ExecStop.
>
> voila, deadlock: systemd waits for ExecStop to finish, ExecStop waits for
> systemd to start daemon to be able to stop it.
>
> Now, I am sure that what I did (use the activation socket as the channel
> over which stop commands are safely sent) is not right but I wonder if
> there is a way to make systemd actually deal with the situation better and
> if not, what I should do instead to deal with someone trying to shutdown
> the daemon without "systemctl stop".
>
> I would be happy to file a bug if needed but I am not really sure the
> problem lies in systemd. Or maybe it is a documentation problem: it would
> help quite a bit if the activation state machine/transitions was documented
> in a human-readable way beyond what can be found by reading the source code.
>
> any help would be welcome,
> Mathieu
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20160422/096bb44a/attachment.html>