[systemd-devel] Reliably waiting for udevd to finish processing triggered events

[Tom Gundersen]

On Fri, Mar 6, 2015 at 9:22 PM, Daniel Drake <drake at endlessm.com> wrote:
> plymouth watches for the creation of drm display devices during boot.
> If it finds one, it starts a graphical splash and that is that.
> However, if the system finishes loading drivers and no drm device is
> available, it falls back onto a fbdev-based splash or a text-based
> boot. Once it has made that choice there is no turning back, it
> basically ignores drm devices if they become available later.

This sounds like a broken model (even if the problems you outlined
below were fixed). There is no reason to believe that all drm devices
will be available by the time udev has finished coldplug. Devices may
be enumerated at any time.

> In order to know when the system has finished loading drivers,
> plymouth does the same as "udevadm settle" - it uses udev API's to
> inotify-monitor /run/udev, and it assumes that when the queue file is
> deleted, all driver load events have been processed. But there seem to
> be a couple of problems associated with this.
>
> Firstly, plymouth does the above when it loads in the initramfs. The
> initramfs will trigger udev events for all devices, but if systemd
> finds the root filesystem before plymouth finds the drm device, udevd
> is immediately killed by systemd as it changes to switch-root.target.
> udevd has not processed the drm device at this point, so
> udev_device_get_is_initialized() returns false when plymouth inquires.
> As udevd is killed, it removes /run/udev/queue in its exit path;
> plymouth sees this and (like udevsettle would) assumes that this
> apparently empty queue means that driver loading is complete. But no
> drm devices are available and initialized, so it falls back to textual
> boot for the rest of boot.

Just to stress this again: the assumptions plymouth makes here are
invalid. It correctly figures out that the udev queue is empty, but
that does not mean udev processed all the events (some may have
failed, or as is the case here, udev might have gotten killed), nor
that the relevant events have even reached udev yet.

> The killing of udev seems heavy-handed here, and the way it removes
> the queue file on exit (without first at least going through the
> already-pending events) seems to kill any possibility of a program
> like udevsettle or plymouth knowing if udev finished loading all
> drivers while the initramfs transitions to the real root.

Yes, this cannot really work.

> Secondly, there is a race during startup. udevd launches and it
> actually removes /run/udev/queue (if it were to exist) in the first
> iteration of the mainloop - even before it checked if any events were
> available to process. Anyway, we would normally expect the queue to be
> empty here, it is only after udevd has started up that systemd then
> goes on to run "udevadm trigger" and generate events for udevd to
> handle.

When udevd starts the event queue is empty, so seems correct to me to
remove the file.

> In the case where plymouth is run from the real root (instead of the
> initramfs), once trigger has exited, systemd starts plymouth, which
> then starts immediately using "udev_queue_get_queue_is_empty()" to do
> the detection described above. If plymouth happens to do that before
> udevd has gotten around to processing the first event generated by
> udevtrigger, the queue is reported as empty (udevd has not created the
> marker yet), so plymouth concludes that driver loading has completed.
> Oops.

Yes, without the ping that udev-settle does, this is racy. Why not
just order plymouth after udev-settle, if that is what you need?

> I believe the same race exists with "udevadm settle", if it is
> launched at that same moment it could hit the same race. The only
> difference is that "udevadm settle" uses some internal udev API that
> actually sends a ping to udevd before it checks the queue status. That
> likely reduces the probability of the race, but I think it is still
> there, as I can't see any guarantee that udevd would create the queue
> file before responding to the ping (it only creates the queue file at
> the start of the next iteration of the main loop, assuming that it had
> noted the pending events in the previous iteration where it also
> handled the ping).

Looks like you are right. I just pushed a patch for this.

> If there's a way of running "udevadm trigger" and then reliably
> knowing that udevd has finished processing those events, I haven't
> found it. Any hints much appreciated.

So ordering yourself after udev-settle should (now) guarantee that
udev has finished processing events caused by udev-trigger (hopefully
processed them all, but some could have failed, or the processing
could have been cancelled). The important thing here is that while
this is often good enough, your kernel may not have enumerated all the
devices yet so using this to decide whether or not you have a drm (or
any other kind of) device can't really work.

Thanks for the report though.

Tom