[systemd-devel] mysqld about back to use mysqld_safe in Fedora

[Lennart Poettering]

On Wed, 27.06.12 14:50, Honza Horak (hhorak at redhat.com) wrote:

> Hi guys,

Heya,

> we've met some problems around mysqld and systemd's auto-restart
> feature recently in Fedora, which has been fortunately fixed by
> Michal few days ago [1]. However, there are still some doubts, which
> convince us to go back and use mysqld_safe script for monitoring and
> restarting potential crashed main process, instead of let it be done
> by systemd.
> 
> One of the issue is (in)ability to imitate the original behavior of
> mysqld_safe, which restarts the main process only if it crashes and
> the pid file isn't properly cleaned up. There's probably not a way
> how to do something like that in systemd, is it? AFAIK we can only
> use option on-abort, but we'd have to be sure that exit code is
> never 0 in cases we want the daemon to be restarted.

Hmm, well, not really. "on-abort" means that systemd would only restart
the service on actual crashes, i.e. where the main process exits with a
signal like SIGSEGV, SIGBUS or SIGABRT. This is different from
"on-failure" which is the behaviour you describe where we'd also restart
the service if it exists with a non-zero exit code.

I am pretty sure "on-abort" is actually what you really want here, no?
The PID file checking sounds like a hack to achieve the same?

> Another issue is the PID file guessing. It usually works fine, but
> if we want to be sure what pid is monitored, we should use PIDFile=
> option. However, when we use this we don't know when the pidfile is
> created, while waiting some sane time is not a nice solution and
> could bring another issues. Using notify feature doesn't helps us,
> since we used Type=forking.

Hmm, does mysql fork on its own? 

Normally a SysV daemon should have a clear initialization phase that it
informs the init system about by double-forking (and returning in the
parent). And the PID file should be written in that initialization
phase, and the parent should return *only* after the PID file has been
sucessfully written. If the daemon writes the PID file only after having
forked away doing anything with the PID file necessarily racy,
regardless whether this is systemd or sysv.

An alternative, more modern way to handle all this is by simply using
Type=notify and invoking sd_notify("READY=1"); in the daemon when
everything is done. This should normally be a two line patch and makes
PID files entirely redundant.

> Another problem is, that since the main process pid is known too
> late, we cannot use it in mysqld-wait-ready, which is a script run
> in ExecStartPost to check if mysqld daemon is ready to accept
> connections.

Humm, all this sounds really broken. MySQL really should not exit in the
parent before it finished writing the PID file. That's that way on SysV,
and on systemd too...

I think the best way to solve this really is to fix MySQL to either only
exit in the parent when initialization is complete, or to invoke
sd_notify() as described above. Then suddenly all problems go away,
mysqld-wait-ready will be unnecessary and mysqld will behave much more
like any other daemon than before?

> There have been another thoughts like socket activation when we were
> working on mysqld unit file a year ago, but this feature seemed to
> be more problematic than helpful. Feel free to go through the
> discussion to read more at [2].
> 
> As written above, we're about to use mysqld_safe to do all the stuff
> around monitoring the main process and changing service to
> Type=simple again. Maybe you're asking why I'm telling you. Well,
> take it as a feedback, what issues we currently have during mysqld
> packaging. And off course, if you have any thoughts that can help,
> we'll be happy to hear that.

We are definitely interested in feedback. There are certainly some
things that we can improve in systemd to handle these things, so this is
much appreciated.

Lennart

-- 
Lennart Poettering - Red Hat, Inc.