[PATCH v2] fbmem: don't allow too huge resolutions

[Geert Uytterhoeven]

On Wed, Sep 1, 2021 at 3:15 AM Tetsuo Handa
<penguin-kernel at i-love.sakura.ne.jp> wrote:
> syzbot is reporting page fault at vga16fb_fillrect() [1], for
> vga16fb_check_var() is failing to detect multiplication overflow.
>
>   if (vxres * vyres > maxmem) {
>     vyres = maxmem / vxres;
>     if (vyres < yres)
>       return -ENOMEM;
>   }
>
> Since no module would accept too huge resolutions where multiplication
> overflow happens, let's reject in the common path.
>
> Link: https://syzkaller.appspot.com/bug?extid=04168c8063cfdde1db5e [1]
> Reported-by: syzbot <syzbot+04168c8063cfdde1db5e at syzkaller.appspotmail.com>
> Debugged-by: Randy Dunlap <rdunlap at infradead.org>
> Signed-off-by: Tetsuo Handa <penguin-kernel at I-love.SAKURA.ne.jp>

Reviewed-by: Geert Uytterhoeven <geert+renesas at glider.be>

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds