[PATCH v21 2/4] compositor-drm: Allow scanout plane to be occluded by overlay

[Daniel Stone]

a0f8276fe814 ("compositor-drm: Disallow overlapping overlay planes") was
a little too pessimistic in rejecting occluded views. Whilst it
correctly prevented overlay planes from occluding each other, it also
prevented overlay planes from occluding the scanout plane.

This is undesirable: the primary/scanout plane is specified to stack
strictly below all overlay planes, so there is no need to reject a plane
from consideration for scanout due to being occluded by an overlay
plane.

Shift the check downwards so it only applies to overlay rather than
scanout planes.

Signed-off-by: Daniel Stone <daniels at collabora.com>
---
 libweston/compositor-drm.c | 23 ++++++++++++-----------
 1 file changed, 12 insertions(+), 11 deletions(-)

diff --git a/libweston/compositor-drm.c b/libweston/compositor-drm.c
index d192eec16..3d1847790 100644
--- a/libweston/compositor-drm.c
+++ b/libweston/compositor-drm.c
@@ -3426,17 +3426,6 @@ drm_output_propose_state(struct weston_output *output_base,
 		if (pixman_region32_not_empty(&surface_overlap))
 			force_renderer = true;
 
-		/* We do not control the stacking order of overlay planes;
-		 * the scanout plane is strictly stacked bottom and the cursor
-		 * plane top, but the ordering of overlay planes with respect
-		 * to each other is undefined. Make sure we do not have two
-		 * planes overlapping each other. */
-		pixman_region32_intersect(&surface_overlap, &occluded_region,
-					  &clipped_view);
-		if (pixman_region32_not_empty(&surface_overlap))
-			force_renderer = true;
-		pixman_region32_fini(&surface_overlap);
-
 		/* The cursor plane is 'special' in the sense that we can still
 		 * place it in the legacy API, and we gate that with a separate
 		 * cursors_are_broken flag. */
@@ -3457,6 +3446,18 @@ drm_output_propose_state(struct weston_output *output_base,
 		 * scanout plane. */
 		if (!ps && !force_renderer && !renderer_ok)
 			ps = drm_output_prepare_scanout_view(state, ev, mode);
+
+		/* We do not control the stacking order of overlay planes;
+		 * the scanout plane is strictly stacked bottom and the cursor
+		 * plane top, but the ordering of overlay planes with respect
+		 * to each other is undefined. Make sure we do not have two
+		 * planes overlapping each other. */
+		pixman_region32_intersect(&surface_overlap, &occluded_region,
+					  &clipped_view);
+		if (pixman_region32_not_empty(&surface_overlap))
+			force_renderer = true;
+		pixman_region32_fini(&surface_overlap);
+
 		if (!ps && !force_renderer)
 			ps = drm_output_prepare_overlay_view(state, ev, mode);
 
-- 
2.17.1