[systemd-devel] How do I remove dead process managed by Systemd?

[vcaputo at pengaru.com]

On Mon, Dec 04, 2017 at 07:14:51PM +0100, Lennart Poettering wrote:
> On Mo, 04.12.17 23:15, Yun-Chih Chen (yunchih.cat at gmail.com) wrote:
> 
> > Hi, fellows:
> > 
> > How do I remove a dead process (D-state process) wrapped as a Systemd
> > service unit?  The process's parent is systemd (pid=1) and I still did not
> > find a way to remove such process without rebooting.  I have tried the
> > following:
> > 
> > $ systemctl restart <my_service>  # timeout
> > $ systemctl daemon-reexec
> > $ kill -9 <pid of my process> # won't work of course
> > 
> > Or anyone knows of hacks that do the trick?
> 
> You cannot kill "D" state processes on Linux. That's what "D" state
> means ultimately: *uninterruptible* sleep. The kill command can't kill
> such processes, and neither can systemd. If you see processes like
> this, then either you have very slow I/O, or this points to some form
> of kernel/driver problem, as sleeping in this state should usually be
> for short periods only.
> 

That's not strictly accurate AFAIK.

Not all D state processes are unkillable.  D state contributes to load
average, so it's not just to represent unkillable tasks.  IIRC it's
ambiguous from userspace by just glancing at D state if the process is
killable, but it can happen that the process is plain uninterruptible.

I don't have time at the moment to do much digging to confirm.  If you
glance at wait.h from the kernel headers, see wait_event_killable().
TASK_KILLABLE includes TASK_UNINTERRUPTIBLE, and I believe that shows as a
D state in userspace, but it also includes TASK_WAKEKILL.

What Lennart describes still holds true for Yun-Chih; if the process is in
an unkillable state then there's nothing systemd can do.

Regards,
Vito Caputo