[systemd-devel] Reliably waiting for udevd to finish processing triggered events

[Daniel Drake]

Hi,

I'm looking at some issues with the plymouth boot splash system, and
why it intermittently fails to get graphics on screen.

plymouth watches for the creation of drm display devices during boot.
If it finds one, it starts a graphical splash and that is that.
However, if the system finishes loading drivers and no drm device is
available, it falls back onto a fbdev-based splash or a text-based
boot. Once it has made that choice there is no turning back, it
basically ignores drm devices if they become available later.

In order to know when the system has finished loading drivers,
plymouth does the same as "udevadm settle" - it uses udev API's to
inotify-monitor /run/udev, and it assumes that when the queue file is
deleted, all driver load events have been processed. But there seem to
be a couple of problems associated with this.

Firstly, plymouth does the above when it loads in the initramfs. The
initramfs will trigger udev events for all devices, but if systemd
finds the root filesystem before plymouth finds the drm device, udevd
is immediately killed by systemd as it changes to switch-root.target.
udevd has not processed the drm device at this point, so
udev_device_get_is_initialized() returns false when plymouth inquires.
As udevd is killed, it removes /run/udev/queue in its exit path;
plymouth sees this and (like udevsettle would) assumes that this
apparently empty queue means that driver loading is complete. But no
drm devices are available and initialized, so it falls back to textual
boot for the rest of boot.

The killing of udev seems heavy-handed here, and the way it removes
the queue file on exit (without first at least going through the
already-pending events) seems to kill any possibility of a program
like udevsettle or plymouth knowing if udev finished loading all
drivers while the initramfs transitions to the real root.


Secondly, there is a race during startup. udevd launches and it
actually removes /run/udev/queue (if it were to exist) in the first
iteration of the mainloop - even before it checked if any events were
available to process. Anyway, we would normally expect the queue to be
empty here, it is only after udevd has started up that systemd then
goes on to run "udevadm trigger" and generate events for udevd to
handle.

In the case where plymouth is run from the real root (instead of the
initramfs), once trigger has exited, systemd starts plymouth, which
then starts immediately using "udev_queue_get_queue_is_empty()" to do
the detection described above. If plymouth happens to do that before
udevd has gotten around to processing the first event generated by
udevtrigger, the queue is reported as empty (udevd has not created the
marker yet), so plymouth concludes that driver loading has completed.
Oops.

I believe the same race exists with "udevadm settle", if it is
launched at that same moment it could hit the same race. The only
difference is that "udevadm settle" uses some internal udev API that
actually sends a ping to udevd before it checks the queue status. That
likely reduces the probability of the race, but I think it is still
there, as I can't see any guarantee that udevd would create the queue
file before responding to the ping (it only creates the queue file at
the start of the next iteration of the main loop, assuming that it had
noted the pending events in the previous iteration where it also
handled the ping).


If there's a way of running "udevadm trigger" and then reliably
knowing that udevd has finished processing those events, I haven't
found it. Any hints much appreciated.

Thanks,
Daniel