[PATCH v20 01/10] compositor-drm: Disallow overlapping overlay planes

[Daniel Stone]

The scanout plane strictly stacks under all overlay planes, and the
cursor plane above. However, the stacking of overlay planes with respect
to each other is undefined.

We can control the stacking order of overlay planes with the zpos
property, though this significantly complicates plane assignment. In the
meantime, simply disallow assigning a view to an overlay, when it
overlaps another view which is already on an overlay. This ensures
stacking order is irrelevant, since the planes never intersect each
other.

Signed-off-by: Daniel Stone <daniels at collabora.com>
Reviewed-by: Pekka Paalanen <pekka.paalanen at collabora.co.uk>
---
 libweston/compositor-drm.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/libweston/compositor-drm.c b/libweston/compositor-drm.c
index 51b2482d9..2305f708f 100644
--- a/libweston/compositor-drm.c
+++ b/libweston/compositor-drm.c
@@ -3324,6 +3324,16 @@ drm_output_propose_state(struct weston_output *output_base,
 					  &clipped_view);
 		if (pixman_region32_not_empty(&surface_overlap))
 			next_plane = primary;
+
+		/* We do not control the stacking order of overlay planes;
+		 * the scanout plane is strictly stacked bottom and the cursor
+		 * plane top, but the ordering of overlay planes with respect
+		 * to each other is undefined. Make sure we do not have two
+		 * planes overlapping each other. */
+		pixman_region32_intersect(&surface_overlap, &occluded_region,
+					  &clipped_view);
+		if (pixman_region32_not_empty(&surface_overlap))
+			next_plane = primary;
 		pixman_region32_fini(&surface_overlap);
 
 		if (next_plane == NULL)
-- 
2.17.1