[systemd-devel] systemd-udevd: Race condition when rule starts both a systemd-mount and an unit accessing that mount

[Andrei Borzenkov]

On Wed, Aug 25, 2021 at 2:26 PM Manuel Wagesreither <ManWag at fastmail.fm> wrote:
>
> Hello all,
>
> this is my first post on this mailing list and, first of all, I'd like to thank you and appreciate your work on systemd in general. I admire the logic, the completeness of the manpages and in general how beautifully things are engineered. I'm no unix graybeard and systemd saved me from having to learn all that legacy stuff systemd replaces. Compared to fstab, /etc/network/interfaces and init.d, systemd is a piece of art.
>
> ---
>
> I'm working on an embedded device which should access and scan connected usb drives for certain files. I seem to witness a race condition with my current solution. I would ask for advice on how to implement this functionality in a better way.
>
> When a device /dev/sdb1 is connected, the udev rule below starts BOTH
> * a systemd-service "start-standalone-mender-deployment at media-sdb1.service"
> * `systemd-mount --no-block --automount=no --options=ro --collect /dev/sdb1 /media/sdb1`
>
> The service then starts a shell script accessing the usb drive. Occasionally, it says the directory the usb drive is mounted at is empty. When checking manually, I see it's not. I strongly suspect the script accessed the directory before the usb drive got mounted there.
>
> Here's the udev rule:
> ```
> ACTION=="add", SUBSYSTEMS=="usb", SUBSYSTEM=="block", KERNEL=="*[0-9]*", ENV{ID_FS_USAGE}=="filesystem", TAG+="systemd", ENV{SYSTEMD_WANTS}+="start-standalone-mender-deployment at media-$name.service", RUN{program}+="/usr/bin/systemd-mount --no-block --automount=no --options=ro --collect $devnode /media/$name"
> ```
>
> And here's the systemd service:
> It is templated and gets instantiated with "media-sdb1". It therefore has an "After=media-sdb1.mount". I suspect Systemd-udevd executes the ENV{SYSTEMD_WANTS} part before the RUN{program} part. Hence, "media-sdb1.mount" doesn't yet exist when the service gets started, as it gets created a tad later by systemd-mount.
>
> ```
> [Unit]
> Description=Start standalone Mender deployment (%i)
> After=%i.mount
>
> [Service]
> Type=oneshot
> Restart=no
> ExecStart=/bin/sh /usr/bin/start-standalone-mender-deployment.sh /%I
> ```
>
> Can you confirm my theory?
>

This sounds reasonable. Yet again the same confusion - dependencies
are between jobs. "After=%i.mount" actually means "wait until start
job for %i.mount completes before selecting start job for your unit".
If the start job for your unit happens to be selected for execution
before the start job for %i.mount was queued (it is quite possible as
systemd-mount invocation takes time), there is no start job to wait
for and After becomes noop.

> The only alternative I see is to invoke systemd-mount without --no-block from the shell script itself. Instead of communicating the mount point (media-sdb1) via unit template parameter, I would communicate the device path (/dev/sdb1) to the template unit and pass it on to the shell script, which would determine mount point based on that.
>

Yes, there is no nice solution. Systemd was not designed for dynamic
dependencies like in your case. While it is possible to add
Wants=%i.mount to your unit, there is no guarantee that this unit
definition will be present (due to the same race condition).

Sometimes in this case a unit definition file is created and
"systemctl daemon-reload" is invoked. This has its own can of worms so
can hardly be recommended.

So your approach is probably the most practical one.

Hmm ... if systemd-mount --property accepts Wants and Before, your
mount unit could pull in your service unit. I cannot test right now.

> I'm all ears if you have comments or advice on that. I guess I'm not the first one implementing something like this.
>
>
> Regards,
> Manuel
>
> P.S.: I won't be able to respond until Sunday Aug 29th.