[PATCH] drm/fbdev-generic: Fix framebuffer on big endian devices

[Geert Uytterhoeven]

Hi Thomas,

On Fri, Jun 28, 2024 at 8:07 AM Thomas Zimmermann <tzimmermann at suse.de> wrote:
> Am 27.06.24 um 19:35 schrieb Thomas Huth:
> > Starting with kernel 6.7, the framebuffer text console is not working
> > anymore with the virtio-gpu device on s390x hosts. Such big endian fb
> > devices are usinga different pixel ordering than little endian devices,
> > e.g. DRM_FORMAT_BGRX8888 instead of DRM_FORMAT_XRGB8888.
> >
> > This used to work fine as long as drm_client_buffer_addfb() was still
> > calling drm_mode_addfb() which called drm_driver_legacy_fb_format()
> > internally to get the right format. But drm_client_buffer_addfb() has
> > recently been reworked to call drm_mode_addfb2() instead with the
> > format value that has been passed to it as a parameter (see commit
> > 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()").
> >
> > That format parameter is determined in drm_fbdev_generic_helper_fb_probe()
> > via the drm_mode_legacy_fb_format() function - which only generates
> > formats suitable for little endian devices. So to fix this issue
> > switch to drm_driver_legacy_fb_format() here instead to take the
> > device endianness into consideration.
> >
> > Fixes: 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()")
> > Closes: https://issues.redhat.com/browse/RHEL-45158
> > Signed-off-by: Thomas Huth <thuth at redhat.com>
>
> Acked-by: Thomas Zimmermann <tzimmermann at suse.de>
>
>
> > ---
> >   drivers/gpu/drm/drm_fbdev_generic.c | 3 ++-
>
> This file is now called drm_fbdev_ttm.c in drm-misc-next. And a similar
> patch might be necessary for drm_fbdev_dma.c. The code in
> drm_fbdev_shmem.c apparently has it already.

We are getting too many copies of this logic...
(yup, had to fix them all up in my WIP support for R1 ;-)

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds