Xfbdev color problems due to endianess

[Leon Ljunggren]

On Wed, Sep 10, 2008 at 3:09 PM, Daniel Stone <daniel at fooishbar.org> wrote:
> On Wed, Sep 10, 2008 at 03:05:48PM +0200, Leon Ljunggren wrote:
>> I have compiled and got kdrive (Xfbdev), using the latest stable
>> source from git, running on my ARM based system but I'm experiencing
>> problems with colours., they are "inverted". This is casued by the
>> fact that my machine is big endian while the gfx module is little
>> endian. This means that:
>>
>> BBBBBGGGGGGRRRRR
>>
>> becomes
>>
>> GGGRRRRRBBBBBGGG
>>
>> In fbdev.c there's a color masking feature (in
>> fbdevScreenInitialize(...), line 264-266), that I thought I could
>> modify to fix the problem:
>>
>> screen->fb[0].redMask = Mask (3, 5);
>> screen->fb[0].greenMask = Mask (0, 3) | Mask (13, 3);
>> screen->fb[0].blueMask = Mask (8, 5);
>>
>> However, when I run with these changes the server quits with a
>> segmentation fault error. It would seem that it doesn't like having
>> the green section split up in two part. I can swap the masks around as
>> I wish as long as I don't try to split the bits for any one mask.
>>
>> Anyone that has any idea on what I can do to fix this? I guess I could
>> swap the byte order of each pixel as they are written to the
>> framebuffer, but I haven't been able to find where this writing is
>> done in the server.
>
> Hi,
> Yeah, it requires a contiguous range.  You can use the wfb module
> (replace all calls to fb with wfb), to handle things like this.  IIRC
> nVidia use it to generate tiled framebuffers on G80+.
>
> Cheers,
> Daniel
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
>
> iEYEARECAAYFAkjHxvgACgkQUVYB1rKAgJSMOQCgqT1rfQvKzdnM51yZvDuvqc2h
> 4ggAnjhD8FKSqMgGvKYMkmFJmRkx1KNK
> =IaIk
> -----END PGP SIGNATURE-----
>
>

I don't understand exactly how I would do this (would wfb even work on
my platform)? Wouldn't it be easier to just swap the byte order for
the pixel before xfbdev writes to my frame buffer device? The only
problem is I can't find where it does this.

/Leon