[systemd-devel] question on special configuration case

[Hebenstreit, Michael]

Thanks for this and the other suggestions!

So for starters we’ll disable logind and dbus, increase watchdogsec and see where the footprint is – before disabling journald if necessary in a next step.

Regards
Michael


From: Mantas Mikulėnas [mailto:grawity at gmail.com]
Sent: Wednesday, June 08, 2016 11:35 AM
To: Hebenstreit, Michael
Cc: Systemd
Subject: Re: [systemd-devel] question on special configuration case

This sounds like you could start by unsetting WatchdogSec= for those daemons. Other than the watchdog, they shouldn't be using any CPU unless explicitly contacted.
On Wed, Jun 8, 2016, 02:50 Hebenstreit, Michael <michael.hebenstreit at intel.com<mailto:michael.hebenstreit at intel.com>> wrote:
The base system is actually pretty large (currently 1200 packages) - I hate that myself. Still performance wise the packages are not the issue. The SSDs used can easily handle that, and library loads are only happening once at startup (where the difference van be measured, but if the runtime is 24h startup time of 1s are not an issue). Kernel is tweaked, but those changes are relatively small.

The single problem biggest problem is OS noise. Aka every cycle that the CPU(s) are working on anything but the application. This is caused by a  combination of "large number of nodes" and "tightly coupled job processes".

Our current (RH6) based system runs with a minimal number of demons, none of them taking up any CPU time unless they are used. Systemd process are not so well behaved. After a few hours of running they are already at a few seconds. On a single system - or systems working independent like server farms - that is not an issue. On our systems each second lost is multiplied by the number of nodes in the jobs (let's say 200, but it could also be up to 10000 or more on large installations) due to tight coupling. If 3 demons use 1s a day each (and this is realistic on Xeon Phi Knights Landing systems), that's slowing down the performance by almost 1% (3 * 200 / 86400 = 0.7% to be exact). And - we do not gain anything from those demons after initial startup!

My worst experience with such issues was on a cluster that lost 20% application performance due to a badly configured crond demon. Now I do not expect systemd to have such a negative impact, but even 1%, or even 0.5% of expected loss are too much in our case.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20160608/f240e800/attachment-0001.html>