[systemd-devel] Support for large applications

[Zbigniew Jędrzejewski-Szmek]

On Wed, Feb 17, 2016 at 02:35:55PM +0200, Avi Kivity wrote:
> We are using systemd to supervise our NoSQL database and are
> generally happy.
> 
> A few things will help even more:
> 
> 1. log core dumps immediately rather than after the dump completes
> 
> A database will often consume all memory on the machine; dumping
> 120GB can take a lot of time, especially if compression is enabled.
> As the situation is now, there is a period of time where it is
> impossible to know what is happening.
> 
> (I saw that 229 improves core dumps, but did not see this specifically)

The coredump is logged afterwards because that's the only way to
include all information (including the compressed file name) in one
log message. But there are two changes which might mitigate the problem:
- semi-recently we switched to lz4, which compresses significantly faster,
  have you tried that?

- recently the responsibility of writing core dumps was split out to
  a service. I'm not sure how that influences the time when the log
  message is written.

> 2. parallel compression of core dumps
> 
> As well as consuming all of memory, we also consume all cpus.  Once
> we dump core we may as well use those cores for compressing the huge
> dump.

This should be implemented in the compression library. The compressor
does not seem to be threaded, but it was we would try to make use of it.
OTOH, single-threaded lz4 is able to produce ~500MB/s of compressed
output, so you'd need a really fast disk to go above that.

> 3. watchdog during startup
> 
> Sometimes we need to perform expensive operations during startup
> (log replay, rebuild from network replica) before we can start
> serving. Rather than configure a huge start timeout, I'd prefer to
> have the service report progress to systemd so that it knows that
> startup is still in progress.

Zbyszek