amdgpu: atomic API and cursor/overlay planes

[Simon Ser]

Hi all,

Recently I've been discussing with various people [1] [2] about amdgpu's
handling of KMS planes. AMD hardware is a bit special when it comes to
the cursor plane, and it's not always 100% clear how that maps with the
KMS API.

Up until now we were using cursor and overlay planes in gamescope [3],
but some changes in the amdgpu driver [1] makes us unable to use planes
(and makes us fallback to composition). Basically, now the overlay
cannot be used at all unless it covers the whole CRTC.

The root cause is the cursor plane. The cursor plane can only be
enabled if there is an unscaled plane covering the whole CRTC right
underneath.

So one might assume that as long as the cursor plane isn't used, the
overlay plane could be enabled even if it doesn't cover the whole CRTC.
Unfortunately, ChromeOS uses the atomic API to setup primary/overlay
planes together with the legacy API to setup the cursor plane. This
means that at any time ChromeOS can toggle the cursor plane and will
expect the cursor plane to work.

For this reason, amdgpu now restricts the use of the overlay plane.
However this penalizes all KMS users because of a single user-space
implementation.

After discussion with Daniel Vetter [2], it seemed reasonable to add
a ChromeOS quirk in the kernel which retains the current behavior, but
require everybody else to use the atomic API properly, ie. not mix
atomic overlay plane usage with legacy cursor plane usage. This would
allow fully atomic user-space to better take advantage of overlay
planes with amdgpu.

Does this seem like an accurate summary of the situation? Does this
seem like a reasonable path forward?

Thanks,

Simon

[1]: https://lore.kernel.org/amd-gfx/SrcUnUUGJquVgjp9P79uV8sv6s-kMHG4wp0S3b4Nh9ksi29EIOye5edofuXkDLRvGfvkkRpQZ9JM7MNqew2B3kFUhaxsonDRXprkAYXaQUo=@emersion.fr/
[2]: https://oftc.irclog.whitequark.org/dri-devel/2021-08-27#1630077257-1630078046;
[3]: https://github.com/Plagman/gamescope