[PATCH v2 (repost)] fbmem: don't allow too huge resolutions

[Daniel Vetter]

On Wed, Sep 08, 2021 at 07:27:49PM +0900, Tetsuo Handa wrote:
> syzbot is reporting page fault at vga16fb_fillrect() [1], for
> vga16fb_check_var() is failing to detect multiplication overflow.
> 
>   if (vxres * vyres > maxmem) {
>     vyres = maxmem / vxres;
>     if (vyres < yres)
>       return -ENOMEM;
>   }
> 
> Since no module would accept too huge resolutions where multiplication
> overflow happens, let's reject in the common path.
> 
> Link: https://syzkaller.appspot.com/bug?extid=04168c8063cfdde1db5e [1]
> Reported-by: syzbot <syzbot+04168c8063cfdde1db5e at syzkaller.appspotmail.com>
> Debugged-by: Randy Dunlap <rdunlap at infradead.org>
> Signed-off-by: Tetsuo Handa <penguin-kernel at I-love.SAKURA.ne.jp>
> Reviewed-by: Geert Uytterhoeven <geert+renesas at glider.be>
> ---
> Changes in v2:
>   Use check_mul_overflow(), suggested by Geert Uytterhoeven <geert at linux-m68k.org>.

Pushed to drm-misc-next-fixes so it should get into current merge window.

I also added a cc: stable here, I htink it's needed.

Thanks a lot to both you&Geert for handling this!
-Daniel

> 
>  drivers/video/fbdev/core/fbmem.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/drivers/video/fbdev/core/fbmem.c b/drivers/video/fbdev/core/fbmem.c
> index 71fb710f1ce3..7420d2c16e47 100644
> --- a/drivers/video/fbdev/core/fbmem.c
> +++ b/drivers/video/fbdev/core/fbmem.c
> @@ -962,6 +962,7 @@ fb_set_var(struct fb_info *info, struct fb_var_screeninfo *var)
>  	struct fb_var_screeninfo old_var;
>  	struct fb_videomode mode;
>  	struct fb_event event;
> +	u32 unused;
>  
>  	if (var->activate & FB_ACTIVATE_INV_MODE) {
>  		struct fb_videomode mode1, mode2;
> @@ -1008,6 +1009,11 @@ fb_set_var(struct fb_info *info, struct fb_var_screeninfo *var)
>  	if (var->xres < 8 || var->yres < 8)
>  		return -EINVAL;
>  
> +	/* Too huge resolution causes multiplication overflow. */
> +	if (check_mul_overflow(var->xres, var->yres, &unused) ||
> +	    check_mul_overflow(var->xres_virtual, var->yres_virtual, &unused))
> +		return -EINVAL;
> +
>  	ret = info->fbops->fb_check_var(var, info);
>  
>  	if (ret)
> -- 
> 2.18.4
> 
> 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch