[systemd-devel] sometimes systemd takes a very long time to restart service

[JB]

Andrey Borzenkov wrote:
> В Wed, 02 Jan 2013 02:28:19 -0700
> JB <general at itpsg.com> пишет:
>
>   
>> Here's the service file:
>> *******************************************
>> [Unit]
>> Description=Webrick Test Service
>> After=network.target
>>
>> [Service]
>> Type=forking
>> ExecStart=/usr/bin/ruby /home/rtuser/test.rb
>>
>> [Install]
>> WantedBy=multi-user.target
>> *******************************************
>>
>> I put this in /etc/systemd/system/webrickd.service
>> Then ran systemctl --system daemon-reload
>> Then ran systemctl start webrickd.service
>>
>> Running it from the command line runs as it should.  Web server starts 
>> and test.rb is running in the background as a daemon returning me back 
>> to the shell.
>>
>> But as soon as I try to use systemd, it starts but as soon as the 
>> WEBrick::Daemon.start call is made, everything is killed.
>>     
> That is because of ...
>   
>> webrickd.service - Webrick Test Service
>>           Loaded: loaded (/etc/systemd/system/webrickd.service)
>>           Active: inactive (dead) since Wed, 02 Jan 2013 01:23:01 -0700; 
>> 9min ago
>>          Process: 1605 ExecStart=/usr/bin/ruby /home/rtuser/test.rb 
>> (code=exited, status=0/SUCCESS)
>>         Main PID: 1607 (code=exited, status=0/SUCCESS)
>>     
>                                ^^^^^^
>   
>>           CGroup: name=systemd:/system/webrickd.service
>>
>>     
>
> You daemon forks too much. This would be a problem with some of
> implementations of initscript as well. Even worse, it may sometimes
> work due to race condition and sometimes it may fail.
>   
As far as I know, it forks once.  You sure it forks "too much?"  I 
should pull an strace just to see exactly how many times it's forking so 
I can be sure.  Who decided how much forking was too much?  Pretty sure 
that's always been a "feature" of unix to be used at the discretion of 
the programmer.  Albeit an expensive one but a feature nonetheless.  Not 
trying to argue but I'm hoping you see my point.  Systemd decided that 
it would by default *only* allow daemon processes to fork once, yet 
accommodate through special configuration (RemainAfterExit) what I would 
consider normal unix programming behavior?  Init doesn't care how many 
times you fork because it's flexible that way.  Now systemd came along 
and decided that was somehow a programming error?
>> I'm really trying to understand systemd but it is times like this when 
>> do long for the simpler days of init.
>>     
>
> initscripts often hide design or programming errors when systemd makes
> them obvious. Do not shoot the messenger.
>   
Not interested in shooting anyone but I am interested in the 
interpretation of "programming error."  Seems like you're assuming the 
systemd assumptions and paradigm are perfect.  Pretty sure Unix never 
dictated default forking limits, at least not until things like fork 
bombs came about and security issues were considered.  Forking more than 
once could hardly be considered a programming error.  I think systemd 
folks came up with that one all by themselves :)  My point is that 
assuming systemd's default allow one fork approach is "pure" and then 
passing the rigid inflexibility off as design flaws and programming 
errors on the part of daemon developers, IMHO goes too far.
>   
>> If someone could tell me the "right way" to make this work with systemd, 
>> I would love to use it but I've been at this on and off for weeks and it 
>> isn't getting any easier.  From my perspective systemd appears rigid and 
>> quite unforgiving.  I can't call it buggy yet because I don't know that 
>> I've found any but it sure doesn't seem nearly as simple and easy to get 
>> something running at startup as the documentation would have me believe.
>>
>>
>>     
>
> Currently systemd assumes "one service - one main process" model. This
> process - main process - represents service for systemd. Service is
> alive as long as it runs. When process exits (or terminates) systemd
> considers service stopped (gracefully or ungracefully - it is another
> matter).
>   
> Systemd also expects that program does what you say it does. When you
> say that service is "simple" it expects that process is started and does
> not fork. If you say that service is "forking" it expects that process
> forks exactly once and child remains. It does not expect that child
> itself forks yet again and exits.
>
>   
That's what I meant when I said I tried "Simple."  I took out all 
daemonize calls.  Started ruby and made just one call to server.start 
using only an instance of WEBrick::HTTPServer (confirming it did not 
form) and it still failed.  Although I just tried it again with the over 
simplified example I had in my previous email and it worked.  I'm now 
going to have to figure out what else this ruby beast is doing that 
systemd doesn't like but if I find out it's another "normally supported" 
feature of Unix that systemd decided is no good it'll be very 
frustrating.  I'm not saying systemd is bad, but rather a little more on 
the inflexible side than I'd prefer.  It seems to force a bit more 
default big-brother type behavior than I like.  Flexibility is one of 
the things that has always drawn me to Unix.

So, if I proceed using "Simple" and then later as a result of one of the 
web service calls made to the server either use fork/exec or system() to 
launch another process that runs for anywhere from 30 seconds to several 
hours, then that process exits but not the one that originally started, 
will systemd have a problem with that?

> In your case obvious workaround is to declare RemainAfterExit=true. In
> this case systemd will not consider service dead as soon as main
> process exited as long as it did not fail (i.e. exit code was 0). 
>   
That may work as well.  I'm fine with Simple and not daemonizing 
anything.  It does simplify my code which is nice, but it won't be worth 
it if it imposes restrictions on what the process can do after it's 
running in order for systemd to stay happy with it.
> Does your daemon need to keep internal state between different HTTP
> requests? If not, you could make it to be socket-activated on demand.
>   
Yes, unfortunately it does.