[PATCH 3/6] drm: fourcc byteorder: add bigendian support to drm_mode_legacy_fb_format

[Eric Engestrom]

On Wednesday, 2017-04-26 07:53:10 +0200, Gerd Hoffmann wrote:
> On Di, 2017-04-25 at 12:18 +0900, Michel Dänzer wrote:
> > On 24/04/17 03:25 PM, Gerd Hoffmann wrote:
> > > Return correct fourcc codes on bigendian.  Drivers must be adapted to
> > > this change.
> > > 
> > > Signed-off-by: Gerd Hoffmann <kraxel at redhat.com>
> > 
> > Just to reiterate, this won't work for the radeon driver, which programs
> > the GPU to use (effectively, per the current definition that these are
> > little endian GPU formats) DRM_FORMAT_XRGB8888 with pre-R600 and
> > DRM_FORMAT_BGRX8888 with >= R600.
> 
> Hmm, ok, how does bigendian fbdev emulation work on pre-R600 then?
> 
> > > +#ifdef __BIG_ENDIAN
> > > +	switch (bpp) {
> > > +	case 8:
> > > +		fmt = DRM_FORMAT_C8;
> > > +		break;
> > > +	case 24:
> > > +		fmt = DRM_FORMAT_BGR888;
> > > +		break;
> > 
> > BTW, endianness as a concept cannot apply to 8 or 24 bpp formats.
> 
> I could move the 8 bpp case out of the #ifdef somehow, but code
> readability will suffer then I think ...

How about something like this?

	uint32_t drm_mode_legacy_fb_format(uint32_t bpp, uint32_t depth)
	{
		uint32_t fmt;
	#ifdef __BIG_ENDIAN
		enum { LITTLE_ENDIAN = 0 };
	#else
		enum { LITTLE_ENDIAN = 1 };
	#endif
	/* ... */

(using an enum for compile-time constness)

and then
	fmt = DRM_FORMAT_ARGB8888;
becomes
	fmt = LITTLE_ENDIAN ? DRM_FORMAT_ARGB8888 : DRM_FORMAT_BGRA8888;

Might be easier to read than duplicating the whole switch?

> 
> For 24 we have different byte orderings, but yes, you can't switch from
> one to the other with byteswapping.  Probably one of the reasons why
> this format is pretty much out of fashion these days ...
> 
> cheers,
>   Gerd
>