[PATCH] drm/fb-helper: Make set_var validation stricter

[Daniel Vetter]

The drm fbdev emulation does not forward mode changes to the driver,
and hence all changes where rejected in 865afb11949e ("drm/fb-helper:
reject any changes to the fbdev").

Unfortunately this resulted in bugs on multiple monitor systems with
different resolutions. In that case the fbdev emulation code sizes the
underlying framebuffer for the largest screen (which dictates
x/yres_virtual), but adjust the fbdev x/yres to match the smallest
resolution. The above mentioned patch failed to realize that, and
errornously validated x/yres against the fb dimensions.

This was fixed by just dropping the validation for too small sizes,
which restored vt switching with 12ffed96d436 ("drm/fb-helper: Allow
var->x/yres(_virtual) < fb->width/height again").

But this also restored all kinds of validation issues and their
fallout in the notoriously buggy fbcon code for too small sizes. Since
no one is volunteering to really make fbcon and vc/vt fully robust
against these math issues make sure this barn door is closed for good
again.

Since it's a bit tricky to remember the x/yres we picked across both
the newer generic fbdev emulation and the older code with more driver
involvement, we simply check that it doesn't change. This relies on
drm_fb_helper_fill_var() having done things correctly, and nothing
having trampled it yet.

Note that this leaves all the other fbdev drivers out in the rain.
Given that distros have finally started to move away from those
completely for real I think that's good enough. The code it spaghetti
enough that I do not feel confident to even review fixes for it.

What might help fbdev is doing something similar to what was done in
a49145acfb97 ("fbmem: add margin check to fb_check_caps()") and ensure
x/yres_virtual aren't too small, for some value of "too small". Maybe
checking that they're at least x/yres makes sense?

v2: Allow more flexibility again for x/yres_virtual (Thomas). Also
print out the visible dimensions we're checking for.

Also actually use the blackhole stable@ address (Greg)

Fixes: 12ffed96d436 ("drm/fb-helper: Allow var->x/yres(_virtual) < fb->width/height again")
Cc: Michel Dänzer <michel at daenzer.net>
Cc: Daniel Stone <daniels at collabora.com>
Cc: Daniel Vetter <daniel.vetter at ffwll.ch>
Cc: Maarten Lankhorst <maarten.lankhorst at linux.intel.com>
Cc: Maxime Ripard <mripard at kernel.org>
Cc: Thomas Zimmermann <tzimmermann at suse.de>
Cc: <stable at kernel.org> # v4.11+
Cc: Helge Deller <deller at gmx.de>
Cc: Greg Kroah-Hartman <gregkh at linuxfoundation.org>
Cc: openeuler-security at openeuler.org
Cc: guodaxing at huawei.com
Cc: Weigang (Jimmy) <weigang12 at huawei.com>
Reported-by: Weigang (Jimmy) <weigang12 at huawei.com>
Signed-off-by: Daniel Vetter <daniel.vetter at intel.com>
---
Note: Weigang asked for this to stay under embargo until it's all
review and tested.
-Daniel
---
 drivers/gpu/drm/drm_fb_helper.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/drm_fb_helper.c b/drivers/gpu/drm/drm_fb_helper.c
index 695997ae2a7c..b6ad8aa9d481 100644
--- a/drivers/gpu/drm/drm_fb_helper.c
+++ b/drivers/gpu/drm/drm_fb_helper.c
@@ -1355,13 +1355,15 @@ int drm_fb_helper_check_var(struct fb_var_screeninfo *var,
 	 * to KMS, hence fail if different settings are requested.
 	 */
 	if (var->bits_per_pixel > fb->format->cpp[0] * 8 ||
-	    var->xres > fb->width || var->yres > fb->height ||
-	    var->xres_virtual > fb->width || var->yres_virtual > fb->height) {
+	    var->xres != info->var.xres || var->yres != info->var.yres ||
+	    var->xres_virtual > fb->width || var->yres_virtual > fb->height ||
+	    var->xres_virtual < var->xres || var->yres_virtual > var->yres) {
 		drm_dbg_kms(dev, "fb requested width/height/bpp can't fit in current fb "
-			  "request %dx%d-%d (virtual %dx%d) > %dx%d-%d\n",
+			  "request %dx%d-%d (virtual %dx%d) > %dx%d-%d (visible %dx%d)\n",
 			  var->xres, var->yres, var->bits_per_pixel,
 			  var->xres_virtual, var->yres_virtual,
-			  fb->width, fb->height, fb->format->cpp[0] * 8);
+			  fb->width, fb->height, fb->format->cpp[0] * 8,
+			  info->var.xres, info->var.yres);
 		return -EINVAL;
 	}
 
-- 
2.36.0