[systemd-devel] question on special configuration case

[Greg KH]

On Tue, Jun 07, 2016 at 11:50:36PM +0000, Hebenstreit, Michael wrote:
> The base system is actually pretty large (currently 1200 packages) - I
> hate that myself. Still performance wise the packages are not the
> issue. The SSDs used can easily handle that, and library loads are
> only happening once at startup (where the difference van be measured,
> but if the runtime is 24h startup time of 1s are not an issue). Kernel
> is tweaked, but those changes are relatively small.
> 
> The single problem biggest problem is OS noise. Aka every cycle that
> the CPU(s) are working on anything but the application. This is caused
> by a  combination of "large number of nodes" and "tightly coupled job
> processes". 

Then bind your applications to the cpus and don't let anything else run
on them, including the kernel.  That way you will not get any jitter or
latencies and can use the CPUs to their max, without having to worry
about anything.  Leave one CPU alone to have the kernel be able to
manage its housekeeping tasks (you seem to be ignoring that issue when
looking at systemd, which is odd to me as it's more noise than anything
else), and also let everything else run there as well.

That what "most" other system designers in your situation do :)

> Our current (RH6) based system runs with a minimal number of demons,
> none of them taking up any CPU time unless they are used. Systemd
> process are not so well behaved. After a few hours of running they are
> already at a few seconds.

What processes are showing up in your count?  Perhaps it's just a bug
that needs to be fixed.

> On a single system - or systems working independent like server farms
> - that is not an issue. On our systems each second lost is multiplied
> by the number of nodes in the jobs (let's say 200, but it could also
> be up to 10000 or more on large installations) due to tight coupling.
> If 3 demons use 1s a day each (and this is realistic on Xeon Phi
> Knights Landing systems), that's slowing down the performance by
> almost 1% (3 * 200 / 86400 = 0.7% to be exact). And - we do not gain
> anything from those demons after initial startup! 

Your kernel is eating more CPU time than those 1s numbers, why you
aren't complaining about that seems strange to me :)

> My worst experience with such issues was on a cluster that lost 20%
> application performance due to a badly configured crond demon.

That's not the issue here though.

Again, what tasks are causing cpu time for "no good reason", let's see
if we can just fix them.

thanks,

greg k-h