[systemd-devel] nspawn COW and instance orchestration

[Umut Tezduyar Lindskog]

Hi,

Regarding orchestration, we’ve done quite a bit of research and previously
explored Nomad as one possible path. Another toolset we recently came
across is Fly.io <https://fly.io/>. Fly works with Docker images, but
instead of running them directly, it extracts the root filesystem from the
image and adds its own init system along with some infrastructure-related
processes. The final image is then booted inside Firecracker microVMs.

The newer generation of Fly machines, along with their updated init
process, now supports PID namespaces. So, in theory, if we package a GPT
root filesystem into a Docker image and set the entrypoint to:
CMD ["unshare", "--pid", "--fork", "--mount-proc", "/lib/systemd/systemd"]
—it should be able to boot on Fly, as suggested by their documentation.

This approach seems somewhat similar to what systemd-nspawn does (perhaps a
bit less), but I’d really appreciate any feedback on potential pitfalls or
considerations with this setup.

Thanks for your time,

Umut

On Tue, Mar 18, 2025 at 6:22 PM Umut Tezduyar Lindskog <umut at tezduyar.com>
wrote:

> Hello,
>
> If multiple instances of the systemd-nspawn are started of the same image,
> do the processes in the images have COW? For example, do PID 1 in both
> instances share the same text section? I couldn't put all the pieces
> together and prove it. I could see that the inode numbers in both instances
> are pointing to the same one (using smaps) but the PSS values didn't
> reflect this.
>
> Somehow unrelated question. Are there mainstream tools that can do
> nspawn orchestration? I stumbled on Nomad in many occasions. My use case is
> to start instances of the same image with systemd-nspawn on a cloud
> cluster, specifically aws. At some point the instance is shut down and
> resources are reused.
>
> Thank you for your time
> Umut
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20250415/690a7215/attachment-0001.htm>