[PATCH 2/3] drm: Make sure at least one plane supports the fb format

[Ville Syrjala]

From: Ville Syrjälä <ville.syrjala at linux.intel.com>

To make life easier for drivers, let's have the core check that the
requested pixel format is supported by at least one plane when creating
a new framebuffer.

This eases the burden on drivers by making sure they'll never get
requests to create fbs with unsupported pixel formats. Thanks to the
new .fb_modifier() hook this check can now be done whether the request
specifies the modifier directly or driver has to deduce it from the
gem bo tiling (or via any other method really).

v0: Accept anyformat if the driver doesn't do planes (Eric)
    s/planes_have_format/any_plane_has_format/ (Eric)
    Check the modifier as well since we already have a function
    that does both
v3: Don't do the check in the core since we may not know the
    modifier yet, instead export the function and let drivers
    call it themselves
v4: Unexport the functiona and put the format_default check back
    since this will again be called by the core, ie. undo v3 ;)

Cc: Eric Anholt <eric at anholt.net>
Testcase: igt/kms_addfb_basic/expected-formats
Signed-off-by: Ville Syrjälä <ville.syrjala at linux.intel.com>
---
 drivers/gpu/drm/drm_framebuffer.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/gpu/drm/drm_framebuffer.c b/drivers/gpu/drm/drm_framebuffer.c
index 21d3d51eb261..e618a6b728d4 100644
--- a/drivers/gpu/drm/drm_framebuffer.c
+++ b/drivers/gpu/drm/drm_framebuffer.c
@@ -152,6 +152,26 @@ static int fb_plane_height(int height,
 	return DIV_ROUND_UP(height, format->vsub);
 }
 
+static bool any_plane_has_format(struct drm_device *dev,
+				 u32 format, u64 modifier)
+{
+	struct drm_plane *plane;
+
+	drm_for_each_plane(plane, dev) {
+		/*
+		 * In case the driver doesn't really do
+		 * planes we have to accept any format here.
+		 */
+		if (plane->format_default)
+			return true;
+
+		if (drm_plane_check_pixel_format(plane, format, modifier) == 0)
+			return true;
+	}
+
+	return false;
+}
+
 static int framebuffer_check(struct drm_device *dev,
 			     struct drm_mode_fb_cmd2 *r,
 			     u64 modifier)
@@ -183,6 +203,16 @@ static int framebuffer_check(struct drm_device *dev,
 		return -EINVAL;
 	}
 
+	if (!any_plane_has_format(dev, r->pixel_format, modifier)) {
+		struct drm_format_name_buf format_name;
+
+		DRM_DEBUG_KMS("unsupported pixel format %s / modifier 0x%llx\n",
+			      drm_get_format_name(r->pixel_format,
+						  &format_name),
+			      modifier);
+		return -EINVAL;
+	}
+
 	for (i = 0; i < info->num_planes; i++) {
 		unsigned int width = fb_plane_width(r->width, info, i);
 		unsigned int height = fb_plane_height(r->height, info, i);
-- 
2.16.1