[systemd-devel] Reason for setting runqueue to IDLE priority and side effects if this is changed?

[Hoyer, Marko (ADITG/SW2)]

Hi all,

jumping from systemd 206 to systemd 211 we were faced with some issue, which are finally caused by a changed main loop priority of the job execution. 

Our use case is the following one:
----------------------------------
While we are starting up the system, a so called application starter is bringing up a set of applications at a certain point in a controlled way by requesting systemd via dbus to start respective units. The reason is that we have to take care for applications internal states as synchronization points and for better meeting timing requirements of certain applications in a generic way. I told the story once before in another post about watchdog observation in state "activating". However ...

Up to v206, the behavior of systemd was the following one:
----------------------------------------------------------
- the starter sends out a start request of a bench of applications (he requests a sequence of unit starts)
- it seems that systemd is working of the sequence exactly as it was requested one by one in the same order

Systemd v211 shows a different behavior:
----------------------------------------
- the starter sends out a bench of requests
- the job execution of the first job is significantly delayed (we have a system under stress, high CPU load at that time)
- suddenly, system starts working of the jobs but now in reverse order
- depending on the situation, it might happen that a complete bench of scheduled jobs are reverse ordered, sometimes two or more sub benches of jobs are executed in the reverse order (the jobs in each bench are reverse ordered)

I found that the system behavior with systemd v206 was only accidently the expected one. The reason was that in this version the run queue dispatching was a fixed part of the main loop located before the dispatching of the events. This way, it gained higher priority than the dbus request handling. One jobs was requested via dbus. Once the dbus job request was dispatched, it's been worked off immediately the next round of the mainloop. Then the next dbus request was dispatched and so on ...

Systemd v211 added the running queue as deferred event source to the event handling with priority IDLE. So dbus requests are preferred over job execution. The reverse order effect is simply because the run queue is more a stack than a queue. All of the observed behavior could be explained I guess.

So long story in advance. I've now two questions:
- Am I causing any critical side effects when I'm increasing the run queue priority so that it is higher than the one of the dbus handling (which is NORMAL)? First tests showed that I can get back exactly the behavior we had before with that.

- Might it still happen that situations are happening where the jobs are reordered even though I'm increasing the priority?

- Is there any other good solution ensuring the order of job execution?

Hope someone has some useful feedback.

Thx in advance.

Marko Hoyer

Advanced Driver Information Technology GmbH
Software Group II (ADITG/SW2)
Robert-Bosch-Str. 200
31139 Hildesheim
Germany

Tel. +49 5121 49 6948
Fax +49 5121 49 6999
mhoyer at de.adit-jv.com

ADIT is a joint venture company of Robert Bosch GmbH/Robert Bosch Car Multimedia GmbH and DENSO Corpoation
Sitz: Hildesheim, Registergericht: Amtsgericht Hildesheim HRB 3438
Geschaeftsfuehrung: Wilhelm Grabow, Katsuyoshi Maeda