Split render/display SoCs, Mesa's renderonly, and Wayland dmabuf hints

[Simon Ser]

Hi all,

I'm working on a Wayland extension [1] that, among other things, allows
compositors to advertise the preferred device to be used by Wayland
clients.

In general, compositors will send a render node. However, in the case
of split render/display SoCs, things get a little bit complicated.

The Wayland compositor will find the display-only DRM device (usually
at /dev/dri/card0). This DRM device will have a DRM primary node, but
no DRM render node.

The Wayland compositor will create a GBM device from this display-only
device, then create an EGL display. Under the hood, Mesa's kmsro will
kick in and magically open a render node from a different device.

However the compositor has no knowledge of this, and has no way to
discover the render node opened by kmsro.

This is an issue because the compositor has no render node to send to
Wayland clients. The compositor is forced to send a primary node to
clients. Clients will need to open the primary node and rely on Mesa's
renderonly to once again magically open the render node under the hood.

In general clients cannot be expected to be able to open primary nodes.
This happens to work on systemd distributions because udev sets a
special uaccess tag on the primary node that makes logind grant
permissions to users physically logged in on a VT.

This will fall apart on non-logind systems and on systems where no user
is physically logged in. Additionally, it may be desirable to deny
access to primary nodes in sandboxes.

So I believe the best way forward would be for the compositor to send
the render node to clients. This could prevent clients to allocate
buffers suitable for scan-out, but that can be fixed with some kind of
buffer constraints negotiation, like we presented at XDC 2020 [2].

There are a few solutions:

1. Require compositors to discover the render device by trying to import
   a buffer. For each available render device, the compositor would
   allocate a buffer, export it as a DMA-BUF, import it to the
   display-only device, then try to drmModeAddFB.
2. Allow compositors to query the render device magically opened by
   kmsro. This could be done either via EGL_EXT_device_drm, or via a
   new EGL extension.
3. Allow compositors to query the kernel drivers to know which devices
   are compatible with each other. Some uAPI to query a compatible
   display device from a render-only device, or vice-versa, has been
   suggested in the past.

(1) has a number of limitations and gotchas. It requires allocating
real buffers, this has a rather big cost for something done at
compositor initialization time. It requires to select a buffer format
and modifier compatible with both devices, so it can't be isolated in
a simple function (and e.g. shared between all compositors in libdrm).
Some drivers will allow to drmModeAddFB buffers that can't be scanned
out, and will only reject the buffer at atomic commit time.

(2) wouldn't work with non-EGL APIs such as Vulkan. Eric Anholt seemed
pretty opposed to this idea, but I didn't fully understood why.

I don't know how feasible (3) is. The kernel drivers must be able to
decide whether buffers coming from another driver can be scanned out,
but how early can they give an answer? Can they give an answer solely
based on a DRM node, and not a DMA-BUF?

Feedback is welcome. Do you agree with the premise that compositors
need access to the render node? Do you have any other potential solution
in mind? Which solution would you prefer?

Thanks,

Simon

[1]: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/8
[2]: https://xdc2020.x.org/event/9/contributions/634/