[systemd-devel] Placing systemd unit files on a non-root partition

[magnus at minimum.se]

Hi!

My team is building a embedded (Linux) platform which is to be distributed
within our company as a binary distribution. On top of the platform, other
teams write applications and put in branding to create products.

To separate the applications from the platform, apps and related files are
placed in a separate partition that gets mounted at boot by systemd.
When the system boots, the applications on the separate partition should
start, preferably by systemd.

TL;DR: How can I make systemd read and start services that are defined by
units located in a directory that becomes available after a certain mount
unit has run in the boot sequence?

The full (long) story of my attempts so far:

Either we place systemd units on the app partition and lets systemd look
there, or we make our own service that just executes everything in a
drop-directory on the app partition (which doesn't sound like the systemd
way..). Below is a list of things I've considered to solve the issue; I'd
love to get your input on it:

1. I've tried making systemd look for units on the app partition, but it
proved more difficult than I anticipated. Since I was unable to change the
systemd search paths, I bind-mounted the app partition into
/usr/local/lib/systemd/system using a mount unit. This "almost" worked,
systemd was able to find units but as soon as I started enabling services,
things went south. I suspect the issue is that enabling meant putting
links in /etc/systemd/system, and when the system boots the bindmount is
setup _after_ links are processed in /etc/systemd/system. The result was
that a unit was working until it was enabled, at which point the service
became "not found" upon the next reboot.

2. To ensure the bindmount is setup earlier, I thought about using a
initramfs. E.g mount the app partition, setup the bindmount to
/usr/local/lib/systemd/system and then switch root and let systemd work.
This would probably work, but as we don't use initramfs in our build
today, it is a major change to our boot process and I'd like to avoid it
if possible.

3. The third option I see is to use generators. I have limited experience
with generators, but it seems to be a mechanism to dynamically add units
as the system boots. I could implement this by storing application
manifests on the application partition and let a generator generate a unit
for each application manifest found. The obvious downside would be that
I'd have to invent some kind of manifest file format that describes how to
start my program, just so that it can converted into a unit file by a
generator. I consider service unit files perfectly capable of describing
such things.

4. A more "far-out" idea is to use systemd --user. Again, I have very
little experience with systemd, but it seems like systemd --user allows
you to run a separate instance of systemd for the purpose of starting user
applications. Is it possible to use systemd --user for my use-case? I.e,
even without any user logged in, launch a bunch of units as a non-root
user. AFAIK, the systemd --user mechanism is typically launched whenever a
user logs in the first time and terminates when the user logs out its last
session. In my case, the user applications must launch at boot and stay
running regardless of users logging in/out. Is systemd --user a viable
option for my scenario?

Thanks in advance,
  Magnus.