[PATCH] drm/fbdev-generic: Fix framebuffer on big endian devices

[Thomas Huth]

On 28/06/2024 08.07, Thomas Zimmermann wrote:
> Hi
> 
> Am 27.06.24 um 19:35 schrieb Thomas Huth:
>> Starting with kernel 6.7, the framebuffer text console is not working
>> anymore with the virtio-gpu device on s390x hosts. Such big endian fb
>> devices are usinga different pixel ordering than little endian devices,
>> e.g. DRM_FORMAT_BGRX8888 instead of DRM_FORMAT_XRGB8888.
>>
>> This used to work fine as long as drm_client_buffer_addfb() was still
>> calling drm_mode_addfb() which called drm_driver_legacy_fb_format()
>> internally to get the right format. But drm_client_buffer_addfb() has
>> recently been reworked to call drm_mode_addfb2() instead with the
>> format value that has been passed to it as a parameter (see commit
>> 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to 
>> drm_mode_addfb2()").
>>
>> That format parameter is determined in drm_fbdev_generic_helper_fb_probe()
>> via the drm_mode_legacy_fb_format() function - which only generates
>> formats suitable for little endian devices. So to fix this issue
>> switch to drm_driver_legacy_fb_format() here instead to take the
>> device endianness into consideration.
>>
>> Fixes: 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to 
>> drm_mode_addfb2()")
>> Closes: https://issues.redhat.com/browse/RHEL-45158
>> Signed-off-by: Thomas Huth <thuth at redhat.com>
> 
> Acked-by: Thomas Zimmermann <tzimmermann at suse.de>
> 
> 
>> ---
>>   drivers/gpu/drm/drm_fbdev_generic.c | 3 ++-
> 
> This file is now called drm_fbdev_ttm.c in drm-misc-next.

Oh, ok, shall I send a v2 that is adjusted to that change, or can it be 
fixed while applying my patch?

> And a similar patch might be necessary for drm_fbdev_dma.c.

Looks similar, indeed. Shall I send a patch for that one, too? ... I 
currently don't have a setup for testing that, though...

  Thomas


> The code in drm_fbdev_shmem.c 
> apparently has it already.
> 
> Best regards
> Thomas
> 
>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/gpu/drm/drm_fbdev_generic.c 
>> b/drivers/gpu/drm/drm_fbdev_generic.c
>> index 97e579c33d84..1e200d815e1a 100644
>> --- a/drivers/gpu/drm/drm_fbdev_generic.c
>> +++ b/drivers/gpu/drm/drm_fbdev_generic.c
>> @@ -84,7 +84,8 @@ static int drm_fbdev_generic_helper_fb_probe(struct 
>> drm_fb_helper *fb_helper,
>>               sizes->surface_width, sizes->surface_height,
>>               sizes->surface_bpp);
>> -    format = drm_mode_legacy_fb_format(sizes->surface_bpp, 
>> sizes->surface_depth);
>> +    format = drm_driver_legacy_fb_format(dev, sizes->surface_bpp,
>> +                         sizes->surface_depth);
>>       buffer = drm_client_framebuffer_create(client, sizes->surface_width,
>>                              sizes->surface_height, format);
>>       if (IS_ERR(buffer))
>