[Mesa-dev] Gallium pixel formats on big-endian

[Adam Jackson]

I've been looking at untangling the pixel format code for big-endian.
My current theory is that blindly byte-swapping values is just wrong.
Data coming from the application is never in the "wrong" order.
Consider the first non-trivial format emitted in u_format_table.c:

    /* taken from an x86 build */
    union util_format_b8g8r8a8_unorm {
       uint32_t value;
       struct {
          uint8_t b;
          uint8_t g;
          uint8_t r;
          uint8_t a;
       } chan;
    };

With little-endian struct packing, 'b' is the least significant byte and
'a' is the most significant.  Therefore the "format name" matches the
memory order representation of the pixel format: b8g8r8a8 means eight
bits of blue in address 0.  I believe this is the intent, and that this
should be preserved independently of either the bit order in the
channels themselves, or the byte order of the host.  Blue always goes at
the low address if B is the first letter in the format, regardless of
how wide B is or what CPU you're on.

When creating the unpack kernel on BE for this format, we byte-swap the
value before channel extraction.  This happens to put the right channel
values in the right bytes; but byte-swapping an r5g6b5 format, or any
other format where channel boundaries are not byte boundaries, would be
wrong.

I think the solution in this case is to simply emit format structs the
other way around on big-endian:

    union util_format_b8g8r8a8_unorm {
       uint32_t value;
       struct {
          uint8_t a;
          uint8_t r;
          uint8_t g;
          uint8_t b;
       } chan;
    };

But, even assuming that approach is correct, I think I still have some
issues.

1) drisw has no way of passing the pixel format of the display to the
renderer. Since I'm reasonably sure that does change between BE and LE
systems, we're effectively hardcoding which kind of machine we'd like to
display on correctly.

2) Is there an additional (bit-)endianness assumption when talking to
hardware?  For B5G6R5, is memory order:

    [b: 0-4] [g: 0-5] [r: 0-4]

or is it:

    [b: 4-0] [g: 5-0] [r: 4-0]

If anyone already knows the answers here, I'd be happy to hear it.

- ajax