[PATCH] drm/doc: device hot-unplug for userspace

Wed May 20 12:55:56 UTC 2020

On Tue, May 19, 2020 at 01:06:49PM +0300, Pekka Paalanen wrote:
> From: Pekka Paalanen <pekka.paalanen at collabora.com>
> 
> Set up the expectations on how hot-unplugging a DRM device should look like to
> userspace.
> 
> Written by Daniel Vetter's request and largely based on his comments in IRC and
> from https://lists.freedesktop.org/archives/dri-devel/2020-May/265484.html .
> 
> Signed-off-by: Pekka Paalanen <pekka.paalanen at collabora.com>
> Cc: Daniel Vetter <daniel at ffwll.ch>
> Cc: Andrey Grodzovsky <andrey.grodzovsky at amd.com>
> Cc: Dave Airlie <airlied at redhat.com>
> Cc: Sean Paul <sean at poorly.run>
> 
> ---
> 
> Disclaimer: I am a userspace developer writing for other userspace developers.
> I took some liberties in defining what should happen without knowing what is
> actually possible or what existing drivers already implement.
> ---
>  Documentation/gpu/drm-uapi.rst | 75 ++++++++++++++++++++++++++++++++++
>  1 file changed, 75 insertions(+)
> 
> diff --git a/Documentation/gpu/drm-uapi.rst b/Documentation/gpu/drm-uapi.rst
> index 56fec6ed1ad8..80db4abd2cbd 100644
> --- a/Documentation/gpu/drm-uapi.rst
> +++ b/Documentation/gpu/drm-uapi.rst
> @@ -1,3 +1,5 @@
> +.. Copyright 2020 DisplayLink (UK) Ltd.
> +
>  ===================
>  Userland interfaces
>  ===================
> @@ -162,6 +164,79 @@ other hand, a driver requires shared state between clients which is
>  visible to user-space and accessible beyond open-file boundaries, they
>  cannot support render nodes.
>  
> +Device Hot-Unplug
> +=================
> +
> +.. note::
> +   The following is the plan. Implementation is not there yet
> +   (2020 May 13).
> +
> +Graphics devices (display and/or render) may be connected via USB (e.g.
> +display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end
> +user is able to hot-unplug this kind of devices while they are being
> +used, and expects that the very least the machine does not crash. Any
> +damage from hot-unplugging a DRM device needs to be limited as much as
> +possible and userspace must be given the chance to handle it if it wants
> +to. Ideally, unplugging a DRM device still lets a desktop to continue
> +running, but that is going to need explicit support throughout the whole
> +graphics stack: from kernel and userspace drivers, through display
> +servers, via window system protocols, and in applications and libraries.
> +
> +Other scenarios that should lead to the same are: unrecoverable GPU
> +crash, PCI device disappearing off the bus, or forced unbind of a driver
> +from the physical device.
> +
> +In other words, from userspace perspective everything needs to keep on
> +working more or less, until userspace stops using the disappeared DRM
> +device and closes it completely. Userspace will learn of the device
> +disappearance from the device removed uevent or in some cases specific
> +ioctls returning EIO.
> +
> +This goal raises at least the following requirements for the kernel and
> +drivers:
> +
> +- The kernel must not hang, crash or oops, no matter what userspace was
> +  in the middle of doing when the device disappeared.
> +
> +- All GPU jobs that can no longer run must have their fences
> +  force-signalled to avoid inflicting hangs to userspace.
> +
> +- KMS connectors must change their status to disconnected.
> +
> +- Legacy modesets and pageflips fake success.
> +
> +- Atomic commits, both real and TEST_ONLY, fake success.
> +
> +- Pending non-blocking KMS operations deliver the DRM events userspace
> +  is expecting.
> +
> +- If underlying memory disappears, the mmaps are replaced with harmless
> +  zero pages where access does not raise SIGBUS. Reads return zeros,
> +  writes are ignored.

Hm this is going to be expensive to implement, every write access would
cause a minor fault to catch it and throw it away. That's not going to be
fast is someone is caught trying to upload a giant texture at that moment
:-/

Could we go with just "Reads and writes will still completely without
errors, but have otherwise undefined behaviour". Then we could go with a
"single shared page for every drmfd" (to avoid leaks), set up with rw
permissions. That's dirt cheap, should be easy to implement and everything
stays fast.

> +
> +- dmabuf which point to memory that has disappeared are rewritten to
> +  point to harmless zero pages, similar to mmaps. Imports still succeed
> +  both ways: an existing device importing a dmabuf pointing to
> +  disappeared memory, and a disappeared device importing any dmabuf.

Same here about mmaps.

> +
> +- Render ioctls return EIO which is then handled in userspace drivers,
> +  e.g. Mesa, to have the device disappearance handled in the way
> +  specified for each API (OpenGL, GL ES: GL_KHR_robustness;
> +  Vulkan: VK_ERROR_DEVICE_LOST; etc.)

I'd go further and spec that driver-specific render ioctl have driver
specific behaviour, but overall must not result in crashes and failures
expect as specified in the client apis you already list.

Some drivers might just go with an uevent and have no errors from ioctls,
some might only want to have an error on a very specific ioctl, some maybe
on all of them. Since no one except the hw specific userspace drivers will
have to deal with that I think best to only specify the outcome we want.

> +
> +Raising SIGBUS is not an option, because userspace cannot realistically
> +handle it.  Signal handlers are global, which makes them extremely
> +difficult to use correctly from libraries like Mesa produces. Signal
> +handlers are not composable, you can't have different handlers for GPU1
> +and GPU2 from different vendors, and a third handler for mmapped regular
> +files.  Threads cause additional pain with signal handling as well.

Hm maybe move that to the entire mmap discussion and why SIGBUS is a
really bad idea?

> +
> +Only after userspace has closed all relevant DRM device and dmabuf file
> +descriptors and removed all mmaps, the DRM driver can tear down its
> +instance for the device that no longer exists. If the same physical
> +device somehow comes back in the mean time, it shall be a new DRM
> +device.

Maybe we should add an explicit note that there's no guarantee about the
new chardev minor this new device will get, it could both inherit the
existing one (you can't open the old one anymore anyway) or get a new one?

Or does userspace want a guarantee, i.e. as long as there's still a handle
open kernel guarantees to not recycle the chardev minor (which is what we
currently do). In that case better to add that to your list of guarantees
above.

I think overall a good start for documenting the uapi expectations of
hotunplug, we'll definitely have to refine this as we go.

Cheers, Daniel

> +
>  .. _drm_driver_ioctl:
>  
>  IOCTL Support on Device Nodes
> -- 
> 2.20.1
> 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch