[syzbot] BUG: unable to handle kernel paging request in vga16fb_fillrect

[Tetsuo Handa]

On 2021/08/30 22:00, Dan Carpenter wrote:
>>> diff --git a/drivers/video/fbdev/vga16fb.c b/drivers/video/fbdev/vga16fb.c
>>> index e2757ff1c23d..e483a3f5fd47 100644
>>> --- a/drivers/video/fbdev/vga16fb.c
>>> +++ b/drivers/video/fbdev/vga16fb.c
>>> @@ -403,7 +403,7 @@ static int vga16fb_check_var(struct fb_var_screeninfo *var,
>>>
>>>         if (yres > vyres)
>>>                 vyres = yres;
>>> -       if (vxres * vyres > maxmem) {
>>> +       if ((u64) vxres * vyres > (u64) maxmem) {
>>
>> Mindlessly changing the sizes is not the solution.
>> Please use e.g. the array_size() helper from <linux/overflow.h>
>> instead.
> 
> On a 64bit system the array_size() macro is going to do the exact same
> casts?  But I do think this code would be easier to understand if the
> integer overflow check were pull out separately and done first:
> 
> 	if (array_size(vxres, vyres) >= UINT_MAX)
> 		return -EINVAL;

This is wrong. array_size() returns ULONG_MAX on 64bits upon overflow and
returns UINT_MAX on 32bits upon overflow. However, UINT_MAX is a valid
value without overflow (e.g. vxres == UINT_MAX / 15 && vyres == 15).
Comparing like "> (u64) UINT_MAX" is to detect only overflow.

array_size() would be helpful for forcing memory allocation to fail
(instead of allocating smaller than actually required).

> 
> 	if (vxres * vyres > maxmem) {
> 		...
> 
> The UINT_MAX is because vxres and vyres are u32.
> 
> This would maybe be the first time anyone ever did an integer overflow
> check like this in the kernel.  It's a new idiom.
> 
> regards,
> dan carpenter
>