[PATCH v3] firmware/sysfb: Fix VESA format selection

[Thomas Zimmermann]

Hi

Am 20.04.23 um 17:57 schrieb Pierre Asselin:
> Some legacy BIOSes report no reserved bits in their 32-bit rgb mode,
> breaking the calculation of bits_per_pixel in commit f35cd3fa7729
> ("firmware/sysfb: Fix EFI/VESA format selection").  However they report
> lfb_depth correctly for those modes.  Keep the computation but
> set bits_per_pixel to lfb_depth if the latter is larger.
> 
> v2 fixes the warnings from a max3() macro with arguments of different
> types;  split the bits_per_pixel assignment to avoid uglyfing the code
> with too many casts.
> 
> v3 fixes space and formatting blips pointed out by Javier, and change
> the bit_per_pixel assignment back to a single statement using two casts.
> 
> Link: https://lore.kernel.org/r/4Psm6B6Lqkz1QXM@panix3.panix.com
> Link: https://lore.kernel.org/r/20230412150225.3757223-1-javierm@redhat.com
> Link: https://lore.kernel.org/dri-devel/20230418183325.2327-1-pa@panix.com/T/#u
> Link: https://lore.kernel.org/dri-devel/20230419044834.10816-1-pa@panix.com/T/#u
> Fixes: f35cd3fa7729 ("firmware/sysfb: Fix EFI/VESA format selection")
> Signed-off-by: Pierre Asselin <pa at panix.com>
> ---
>   drivers/firmware/sysfb_simplefb.c | 13 ++++++++-----
>   1 file changed, 8 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/firmware/sysfb_simplefb.c b/drivers/firmware/sysfb_simplefb.c
> index 82c64cb9f531..6f7c5d0c5090 100644
> --- a/drivers/firmware/sysfb_simplefb.c
> +++ b/drivers/firmware/sysfb_simplefb.c
> @@ -51,15 +51,18 @@ __init bool sysfb_parse_mode(const struct screen_info *si,
>   	 *
>   	 * It's not easily possible to fix this in struct screen_info,
>   	 * as this could break UAPI. The best solution is to compute
> -	 * bits_per_pixel here and ignore lfb_depth. In the loop below,
> +	 * bits_per_pixel from the color bits, reserved bits and
> +	 * reported lfb_depth, whichever is highest.  In the loop below,
>   	 * ignore simplefb formats with alpha bits, as EFI and VESA
>   	 * don't specify alpha channels.
>   	 */
>   	if (si->lfb_depth > 8) {
> -		bits_per_pixel = max(max3(si->red_size + si->red_pos,
> -					  si->green_size + si->green_pos,
> -					  si->blue_size + si->blue_pos),
> -				     si->rsvd_size + si->rsvd_pos);
> +		/* max() macros args should be of the same type */
> +		bits_per_pixel = max3((u16)max3(si->red_size + si->red_pos,
> +					si->green_size + si->green_pos,
> +					si->blue_size + si->blue_pos),
> +				(u16)(si->rsvd_size + si->rsvd_pos),
> +				si->lfb_depth);

I found this casting mess even more unreadable. I went back to v2, fixed 
the style issues and committed the patch as v4 (still under your name).

 
https://cgit.freedesktop.org/drm/drm-tip/commit/?id=1b617bc93178912fa36f87a957c15d1f1708c299

Thanks a lot for the bugfix.

Best regard
Thomas

>   	} else {
>   		bits_per_pixel = si->lfb_depth;
>   	}
> 
> base-commit: 6a8f57ae2eb07ab39a6f0ccad60c760743051026

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Frankenstrasse 146, 90461 Nuernberg, Germany
GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman
HRB 36809 (AG Nuernberg)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/dri-devel/attachments/20230421/0b97ba93/attachment.sig>