[systemd-devel] systemd-udevd: Race condition when rule starts both a systemd-mount and an unit accessing that mount

[Colin Guthrie]

Andrei Borzenkov wrote on 25/08/2021 13:44:
> On Wed, Aug 25, 2021 at 2:26 PM Manuel Wagesreither <ManWag at fastmail.fm> wrote:
>>
>> Hello all,
>>
>> this is my first post on this mailing list and, first of all, I'd like to thank you and appreciate your work on systemd in general. I admire the logic, the completeness of the manpages and in general how beautifully things are engineered. I'm no unix graybeard and systemd saved me from having to learn all that legacy stuff systemd replaces. Compared to fstab, /etc/network/interfaces and init.d, systemd is a piece of art.
>>
>> ---
>>
>> I'm working on an embedded device which should access and scan connected usb drives for certain files. I seem to witness a race condition with my current solution. I would ask for advice on how to implement this functionality in a better way.
>>
>> When a device /dev/sdb1 is connected, the udev rule below starts BOTH
>> * a systemd-service "start-standalone-mender-deployment at media-sdb1.service"
>> * `systemd-mount --no-block --automount=no --options=ro --collect /dev/sdb1 /media/sdb1`
>>
>> The service then starts a shell script accessing the usb drive. Occasionally, it says the directory the usb drive is mounted at is empty. When checking manually, I see it's not. I strongly suspect the script accessed the directory before the usb drive got mounted there.
>>
>> Here's the udev rule:
>> ```
>> ACTION=="add", SUBSYSTEMS=="usb", SUBSYSTEM=="block", KERNEL=="*[0-9]*", ENV{ID_FS_USAGE}=="filesystem", TAG+="systemd", ENV{SYSTEMD_WANTS}+="start-standalone-mender-deployment at media-$name.service", RUN{program}+="/usr/bin/systemd-mount --no-block --automount=no --options=ro --collect $devnode /media/$name"
>> ```
>>
>> And here's the systemd service:
>> It is templated and gets instantiated with "media-sdb1". It therefore has an "After=media-sdb1.mount". I suspect Systemd-udevd executes the ENV{SYSTEMD_WANTS} part before the RUN{program} part. Hence, "media-sdb1.mount" doesn't yet exist when the service gets started, as it gets created a tad later by systemd-mount.
>>
>> ```
>> [Unit]
>> Description=Start standalone Mender deployment (%i)
>> After=%i.mount
>>
>> [Service]
>> Type=oneshot
>> Restart=no
>> ExecStart=/bin/sh /usr/bin/start-standalone-mender-deployment.sh /%I
>> ```
>>
>> Can you confirm my theory?
>>
> 
> This sounds reasonable. Yet again the same confusion - dependencies
> are between jobs. "After=%i.mount" actually means "wait until start
> job for %i.mount completes before selecting start job for your unit".
> If the start job for your unit happens to be selected for execution
> before the start job for %i.mount was queued (it is quite possible as
> systemd-mount invocation takes time), there is no start job to wait
> for and After becomes noop.
> 
>> The only alternative I see is to invoke systemd-mount without --no-block from the shell script itself. Instead of communicating the mount point (media-sdb1) via unit template parameter, I would communicate the device path (/dev/sdb1) to the template unit and pass it on to the shell script, which would determine mount point based on that.
>>
> 
> Yes, there is no nice solution. Systemd was not designed for dynamic
> dependencies like in your case. While it is possible to add
> Wants=%i.mount to your unit, there is no guarantee that this unit
> definition will be present (due to the same race condition).
> 
> Sometimes in this case a unit definition file is created and
> "systemctl daemon-reload" is invoked. This has its own can of worms so
> can hardly be recommended.
> 
> So your approach is probably the most practical one.
> 
> Hmm ... if systemd-mount --property accepts Wants and Before, your
> mount unit could pull in your service unit. I cannot test right now.


Not sure how well it works when mount points are involved but would a 
.path unit work here?

e.g. move the start of the 
start-standalone-mender-deployment at media-sdb1.service out of the udev 
rule and instead worry about the mount path, not the device name?

So the udev rule just worries about mounting, then the path unit 
triggers the start of the mender deployment?

Just a thought and not 100% how well it works with mount points as I 
said as I've really not played much with them.

Col

-- 

Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
   Tribalogic Limited http://www.tribalogic.net/
Open Source:
   Mageia Contributor http://www.mageia.org/
   PulseAudio Hacker http://www.pulseaudio.org/
   Trac Hacker http://trac.edgewall.org/