[systemd-devel] GuessMainPID=no required to make daemon reload work

[Lennart Poettering]

On Mon, 19.05.14 14:35, Brandon Black (blblack at gmail.com) wrote:

> On Fri, May 16, 2014 at 9:38 AM, Lennart Poettering
> <lennart at poettering.net>wrote:
> 
> > On Thu, 15.05.14 20:57, Brandon Black (blblack at gmail.com) wrote:
> >
> > > On Thu, May 15, 2014 at 6:40 PM, Lennart Poettering
> > > <lennart at poettering.net>wrote:
> > > >
> > > > But again, it's generally not a good idea to keep file locks for a
> > > > longer period of time, much as with mutexes... You don't want to make
> > > > other apps which try to get an atomic view on the file hang for long.
> > >
> > > If they don't want to hang, they don't have to.  There are nonblocking
> > > calls to check for a conflicting advisory fcntl() lock.  The pidfile
> > should
> > > be private from random software anyways, other than the daemon itself and
> > > perhaps whatever tools or init system are managing start/stop.
> >
> > Note that other programs might read your PID files too, for example
> > systemd. And systemd won't take a lock on it (we don't want to make PID1
> > deadlock...).
> 
> 
> I guess whether systemd or another init system needs to read the PID file
> is a design question, and my daemon design for this stuff tends to be
> "different".  At least in my model, there's no reason for a normal init
> system to need to read the pidfile - it's only used for exclusivity between
> multiple instances of the same daemon to avoid races, and the fcntl() lock
> information is used by status/stop/restart -like commands to identify a
> running daemon without races or process misidentification.  Exactly one
> daemon holds the fcntl writelock on the designated pidfile pathname, and
> its PID is retrievable by reading the lock, which is released on daemon
> shutdown or death.  And again, under systemd this can be made to work with
> sd_notify of MAINPID.

systemd reads the PID file since we need to know which process is the
"main" process of the daemon, so that we can follows its lifecycle,
collect its return value, show all this to the user, and so on ...

The PID file stuff is kinda API of UNIX since a long time, this is for
example visible in the fact that the old fedora/redhat chconfig init
script header exposed the path to the file.

> Hence you need to make sure you write those files
> > atomically anyway, and you cannot use locks for that, you have to use
> > atomic renames... And if you do that, the locks become pretty useless...
> >
> 
> Atomic renames (+ fsync!) get you a pidfile that's always readable by other
> processes, but what they don't get you is solving the problem of races
> between multiple instances of the daemon itself.  The data contained within
> won't be reliable in a lot of real-world corner cases.  What happens when
> someone issues a start command while another start command was still
> executing?  Or a start while a stop was still executing, etc?  Almost every
> scheme for pidfile management that tries to solve all the issues
> surrounding it (guaranteeing a lack of two conflicting instances of the
> same daemon, not losing track of a daemon by letting another instance
> overwrite its pidfile while it's still running, "stop" commands never try
> to mess with an unrelated process, and that status commands aren't confused
> by them, etc) that I've ever seen has serious flaws.  I still think the
> fcntl() method is the least-flawed of reasonably-portable methods.

Well, PID file management is awful, and so is file locking on UNIX. No
doubt... That's why we make it unnecessary on systemd systems...

> IMHO, given that systemd replaces the init infrastructure for managing
> long-running daemons, and given that it places a lot of practical and
> conceptual limitations on what a daemon is and how it is managed versus a
> traditional semi-portable *nixy way of doing things due to making very
> large changes in this area, I think discussion of best practices for
> portable daemons and how to best integrate those practices to accomplish
> the same goals under systemd is *very* germane to this list.

My recommendation would be to avoid any file locks. Instead write the
PID file into a temporary file, which you then link() to the right
place. THis will fail (on all UNIXes) if the PID file already
exists. Then, leave it to the distros to deal with left-over PID
files. This way, you have a race-free way to create the lock file, so
that at no time the file will be "half-written". Of course, if the
daemon or system carshes (and /var/run was persistent), you might fail
to start, but well, the thing crashed, and removing the PID file an that
case manually, is probably not too much to ask. THis put together sounds
like a reasonably safe thing to do, that works well enough on classic
UNIX, and is completely in line with what systemd expects. On systemd
systems even the left-over PID file issue goes away, since we unlink PID
files when we notice that a daemon goes down.

I hope this makes some sense...

Lennart

-- 
Lennart Poettering, Red Hat