[systemd-devel] Systemd killed processes of custom services instead of graceful shutdown
Kamal Rathi
kr30april at gmail.com
Thu Jun 18 07:07:34 UTC 2020
>- ExecStop command returns before application is actually stopped.
>ExecStop must be synchronous, systemd assumes service is stopped when
>ExecStop command completes
Correct, and yes i finally figure out what service i have to make
grid and rdbms dependent so that applicaiton shutdown will be graceful
>- stopping takes longer then TimeoutStopSec
yes i have put a Timeout and it is working correct
>- your actual database it still started outside of rdbms.service. The
>only process shown in your output is listener, but I would expect rather
>more of Oracle there
database was started by grid.service because when it sees that it was
kill by my processes then it will start those services,
but that part is solved
Now I am more eager to know if is it possible to run anything before it
kills users processes at shutdown / reboot / halt.
or maybe i can be wrong but when shutdown / reboot initiates , it seems it
sends a kill signal to all processes parallely.
I tried to capture ps -efj when shutdown initiates and I have seen few bash
sessions get into defunct state(systemd-user-sessions stop [maybe this has
been called]).
After reading man pages of systemd, as shutdown is being initiated control
is being passed from PID 1 to /usr/lib/systemd/systemd-shutdown.
Can I create any dependency in my custom service so that before sending
kill signals to processes or stopping users sessions scope at shutdown /
reboot / halt / poweroff?
Thanks a lot for such advice and I have clear my doubts towards systemd
with all discussions.
On Wed, Jun 17, 2020 at 12:23 AM Andrei Borzenkov <arvidjaar at gmail.com>
wrote:
> 17.06.2020 07:46, Kamal Rathi пишет:
> >> Please take at least some efforts to format mail so that it can be
> >> properly understood. Otherwise I am afraid your mails will simply be
> >> ignored.
> >
> > I am sorry for inconvenience, i will try to follow correct
> approach
> > afterwards while replying
> >
> >> How RemainAfterExit affects *startup* of services?
> >
> > The services worked as expected and as i fixed the issue where it
> was
> > using su in script.
> > As in background there was SysVinit script which was getting
> invoked
> > when system boot and makes processes falls in users.slice
> > So the startup is working perfectly after disabling that SysVinit
> > script.
> >
> >> So I was right. Your programs are running as user services or part of
> >> user sessions.
> >
> > You were absolutely right, I tried to dig into that and find out as
> it
> > was using internally su,
> > but now only problem remains is shutdown order.
> > As I have created a dependency that rdbms.service should start after
> > grid.service which works perfect at boot.
> > But when I do shutdown / reboot both services stop simultaneously.
> > which i want my grid.service to wait first before rdbms.service will
> > stop
> > As Type mentioned in services are oneshot.
> >
> >> dmesg output does not contain any systemd log. Anyway, we already know
> >> that your services are not really system services.
> >
> > So my service is system services, as let me attach cgls output
> >
> > └─system.slice
> > ├─rdbms.service
> > │ └─8039 /sarahn9db/db/11.2.0/bin/tnslsnr sarahn9 -inherit
> > ├─grid.service
> > │ ├─ 5316 /grid/gi/11.2.0/bin/ohasd.bin reboot
> > │ ├─ 6159 /grid/gi/11.2.0/bin/cssdagent
> >
>
> The obvious possibilities
>
> - ExecStop command returns before application is actually stopped.
> ExecStop must be synchronous, systemd assumes service is stopped when
> ExecStop command completes
>
> - stopping takes longer then TimeoutStopSec
>
> - your actual database it still started outside of rdbms.service. The
> only process shown in your output is listener, but I would expect rather
> more of Oracle there
>
> Without logs it is impossible to say anything more specific.
>
>
> > On Tue, Jun 16, 2020 at 12:23 AM Andrei Borzenkov <arvidjaar at gmail.com>
> > wrote:
> >
> >> 16.06.2020 07:36, Kamal Rathi пишет:
> >>> The fact that you need RemainAfterExit at all hints that processes that
> >>> belong to your service are not running as part of service control
> group.
> >>> Knowing how Oracle has traditionally been managed, I suspect that you
> >>> perform "su - oracle_owner" or similar to start them in which case all
> >>> actual service processes become part of respective user sessions, and
> >>> not part of your system services. There is no way to synchronize
> >>> stopping of processes/services belonging to different users. They are
> >>> completely independent and shutdown for all sessions is initiated in
> >>> parallel.
> >>>
> >>
> >> Please take at least some efforts to format mail so that it can be
> >> properly understood. Otherwise I am afraid your mails will simply be
> >> ignored.
> >>
> >>> If my theory is correct, the fix would be to actually run your systemd
> >>> services as systemd services. If my theory is wrong, provide full from
> >>> system boot to shutdown where it could be seen how your services are
> >>> started/stopped. Enabling systemd debug log level when doing it
> >>> certainly won't harm.
> >>>
> >>>
> >>>
> >>> I have incorporated Type=oneshot and remainAfterExit=yes as it was
> >> helpful
> >>> in startup of services,
> >>
> >>
> >> How RemainAfterExit affects *startup* of services?
> >>
> >>> but still grid.service are starting up in
> >>> user.slice
> >>
> >> So I was right. Your programs are running as user services or part of
> >> user sessions.
> >>
> >>> and shutdown was giving the same results
> >>> I am attaching debug logs .
> >>>
> >>
> >> dmesg output does not contain any systemd log. Anyway, we already know
> >> that your services are not really system services.
> >>
> >>> On Mon, Jun 15, 2020 at 10:34 AM Andrei Borzenkov <arvidjaar at gmail.com
> >
> >>> wrote:
> >>>
> >>>> 15.06.2020 11:01, Kamal Rathi пишет:
> >>>>> Hi Team,
> >>>>>
> >>>>> I have two services which are dependent on each other and are working
> >>>> fine
> >>>>> at boot up but at shutdown / reboot , the processes get killed as
> >>>> shutdown
> >>>>> got initated.
> >>>>>
> >>>>> Services are running fine in particular order but processes got
> killed
> >> .I
> >>>>> have enabled lingering on both users and changed confgiuration in
> >>>>> logind.conf to KillUserProcesses=no but still issue is same
> >>>>>
> >>>>
> >>>> Lingering/KillUserProcesses are relevant only for user
> services/sessions
> >>>> and so far there was no indication you use either.
> >>>>
> >>>>> ##############
> >>>>> Systemd service files content are below
> >>>>>
> >>>>> cat /etc/systemd/system/grid.service
> >>>>> [Unit]
> >>>>> Description=Service to auto start Oracle ASM application
> >>>>> Before=rdbms.service
> >>>>> After=syslog.target network.target nfs-mountd.service autofs.service
> >>>>> systemd-user-sessions.service system.slice
> >>>>> [Service]
> >>>>> Type=simple
> >>>>> TimeoutSec=5min
> >>>>> User=grid
> >>>>> Group=dba
> >>>>> ExecStart=/opt/admin/bin/asm
> >>>>> ExecStop=/opt/admin/bin/asm_stop
> >>>>> RemainAfterExit=yes
> >>>>> [Install]
> >>>>> WantedBy=multi-user.target
> >>>>>
> >>>>>
> >>>>>
> >>>>> cat /etc/systemd/system/rdbms.service
> >>>>> [Unit]
> >>>>> Description=Service to auto start Oracle RDBMS application
> >>>>> Requires=grid.service
> >>>>> After=grid.service syslog.target network.target nfs-mountd.service
> >>>>> autofs.service systemd-user-sessions.service system.slice
> >>>>> [Service]
> >>>>> Type=simple
> >>>>> TimeoutSec=5min
> >>>>> User=osarahn9
> >>>>> Group=dba
> >>>>> ExecStart=/opt/admin/bin/rdbms
> >>>>> ExecStop=/opt/admin/bin/rdbms_stop
> >>>>> RemainAfterExit=yes
> >>>>> [Install]
> >>>>> WantedBy=multi-user.target grid.service
> >>>>>
> >>>>>
> >>>>> let me know if my configuration is faulty or what I have missed so
> that
> >>>>> shutdown should be graceful for services and processes will be
> >>>>> shutdown with systemd custom service?
> >>>>>
> >>>>
> >>>> You do not provide enough information (full logs would be certainly
> much
> >>>> more useful than long description) so I can only give educated guess.
> >>>>
> >>>>> I want first rdbms.service should be called and get process stopped
> >>>> before
> >>>>> grid.services (it seems systemd are killing user.slices processes)
> and
> >> in
> >>>>> startup-inverse should be followed .
> >>>>> Please help .
> >>>>>
> >>>>
> >>>> The fact that you need RemainAfterExit at all hints that processes
> that
> >>>> belong to your service are not running as part of service control
> group.
> >>>> Knowing how Oracle has traditionally been managed, I suspect that you
> >>>> perform "su - oracle_owner" or similar to start them in which case all
> >>>> actual service processes become part of respective user sessions, and
> >>>> not part of your system services. There is no way to synchronize
> >>>> stopping of processes/services belonging to different users. They are
> >>>> completely independent and shutdown for all sessions is initiated in
> >>>> parallel.
> >>>>
> >>>> If my theory is correct, the fix would be to actually run your systemd
> >>>> services as systemd services. If my theory is wrong, provide full from
> >>>> system boot to shutdown where it could be seen how your services are
> >>>> started/stopped. Enabling systemd debug log level when doing it
> >>>> certainly won't harm.
> >>>> _______________________________________________
> >>>> systemd-devel mailing list
> >>>> systemd-devel at lists.freedesktop.org
> >>>> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
> >>>>
> >>>
> >>>
> >>> _______________________________________________
> >>> systemd-devel mailing list
> >>> systemd-devel at lists.freedesktop.org
> >>> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
> >>>
> >>
> >> _______________________________________________
> >> systemd-devel mailing list
> >> systemd-devel at lists.freedesktop.org
> >> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
> >>
> >
> >
> > _______________________________________________
> > systemd-devel mailing list
> > systemd-devel at lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/systemd-devel
> >
>
> _______________________________________________
> systemd-devel mailing list
> systemd-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20200618/a9b82bdd/attachment-0001.htm>
More information about the systemd-devel
mailing list