[systemd-devel] systemctl status and service unit configuration with startup script that exits

[Jonathan de Boyne Pollard]

Owens, Stephen:

> The start script calls other scripts to start two application 
> components and then the start script exits.
>
> One of the application components also spawns a couple of child processes.
>
> There ends up being two pid files, one for each component, describing 
> the state of the application.
>

You have two services.


Owens, Stephen:

> Environment=JAVA_HOME=/opt/xxx/jre
>
> ExecStart=/bin/bash -l /opt/xxx/bin/startApp.sh
>
> ExecStop=-/bin/bash -lc "/opt/xxx/bin/stopApp.sh -np"
>

Java, again.  Java is for some reason a magnet for this sort of stuff.

* http://jdebp.eu./FGA/systemd-house-of-horror/


Owens, Stephen:

> Is there some way I can have systemd monitor the component pids to 
> show ACTIVE status?

You have two services.  Make two service units.  Forget the nutty 
startApp and stopApp scripts.  When you look into them you'll almost 
certainly find that they are a process supervisor in shell script, 
written badly as such things always are.  The systemd House of Horror 
has examples of running Java programs directly from service units (It 
isn't hard.) without the intermediate badly written shell script process 
supervisors.  Stack Exchange has stuff on this, too, and more besides.

* http://unix.stackexchange.com/a/320319/5132

* http://unix.stackexchange.com/a/245038/5132

* http://unix.stackexchange.com/a/322778/5132

* http://unix.stackexchange.com/a/229525/5132

* http://unix.stackexchange.com/a/324286/5132

* http://unix.stackexchange.com/a/321716/5132

* http://unix.stackexchange.com/a/280399/5132


Owens, Stephen:

> I'm stuck with 3rd party startup scripts.
>

No, you are not.  This is just a belief and a self-imposed restriction, 
and it is groundless when it comes down to it.  Look into it, as other 
people have, and you'll find the world is quietly, and finally, coming 
around to the daemontools idea, that has been around since 1996, of 
letting the service manager do this.  The MariaDB world, for example, 
has finally officially let go of its wrapper process supervisor shell 
scripts and documented how to just run the daemon directly under a 
service manager, 13 years after daemontools users asked for this.

* http://jdebp.eu./Softwares/nosh/mariadb-and-mysql.html#Prompt

The irony is that if you actually look at your scripts that start/stop 
two services you will probably find that they are wrappers for wrappers, 
like the startdb.sh and stopdb.sh that these people wrote:

* http://itdadao.com/articles/c15a369637p0.html

* 
https://mohamedazar.com/2015/08/27/oracle-applications-r12-auto-start-on-linux/

Even if you hang on to the incorrect belief that these wrapper scripts 
are set in stone, there's a glaringly obvious decomposition into two 
parts in such cases.  You have two services.

Whenever you see the patterns ...

> ExecStart=/opt/example/bin/something.sh start
> ExecStop=/opt/example/bin/something.sh stop

or

> ExecStart=/opt/example/bin/startSomething.sh
> ExecStop=/opt/example/bin/stopSomething.sh

... the Oracle camelCase and the "stop"/"start" arguments are strong 
indicators that in fact those scripts will turn out to be process 
supervisors written badly in shell script, grepping the process tree, 
using PID files, doing their own service restarts with while loops and 
wait, creating private log files that only ever expand indefinitely, and 
running the actual service processes as a grandchild or even a 
great-grandchild of the main process.  (Some of the most stupid cases 
are when one strips away one unnecessary shell script badly written 
process supervisor, only to find that the process that it is supervising 
is itself *another* shell script badly written process supervisor.)

Another giveaway is the "I had to add RemainAfterExit=yes because 
otherwise it didn't work." as exemplified in the ridiculous edifice 
(that clearly hasn't been tested otherwise the invalid attempts at 
redirection would have been spotted, like they were in three of the 
Stack Exchange questions mentioned earlier) recommended by Oracle 
consultant Tim Hall here:

* 
https://oracle-base.com/articles/linux/linux-services-systemd#creating-linux-services

M. Hall had to add RemainAfterExit=yes because xe is running shell 
scripts that in turn fork and execute programs such as "lsnrctl" and 
"sqlplus", some of which in turn fork and execute further processes and 
exit the parent processes.  The shell scripts aren't necessary at all.  
Xe actually has two services, just like you.  In contrast: this person 
has, at least (the "something start/stop" layer that forks still being 
there), decomposed having two services into having two service definitions:

* https://server-world.info/en/note?os=CentOS_7&p=oracle12c&f=6

If one has to add RemainAfterExit=yes in order to bodge things, then its 
quite likely that one has built such a ridiculous edifice, too. The 
shell scripts are forking subprocesses to run non-builtin commands, 
which are in turn probably *more* shell scripts that *also* fork, and 
exiting the parent process.  None of this is actually necessary, and it 
is just cargo-cult design that gets one into this state.

You do not need reams and multiple levels of intermediary shell scripts 
between a (proper) service manager and a Java program. Really, you 
don't.  It is not a rule.  Java is not as incompetent as this is making 
it out to be.