wl_shm formats on big-endian

[Pekka Paalanen]

Hi,

this is a re-send of a part of "Re: [PATCH weston 01/68] libweston: Add
pixel-format helpers" which I believe deserves its own thread.

The core question is, what do WL_SHM_FORMAT_ARGB888 and
WL_SHM_FORMAT_XRGB888 formats mean on big-endian?

There is a possible clash with Cairo.

As summary, format definitions are written as:

Cairo formats: machine-endian
Pixman formats: machine-endian
DRM formats: little-endian
GBM formats: little-endian... BO formats?
GL formats: machine-endian or sequence of bytes
wl_shm formats: ???(*)


*: Presumably all wl_shm formats copied from DRM formats are
little-endian.

See below for details.

On Tue, 14 Feb 2017 12:30:25 +0200
Pekka Paalanen <ppaalanen at gmail.com> wrote:

> On Mon, 13 Feb 2017 18:24:58 +0000
> Daniel Stone <daniel at fooishbar.org> wrote:
> 

> > >> +     {
> > >> +             .format = DRM_FORMAT_XRGB8888,
> > >> +             .depth = 24,
> > >> +             .bpp = 32,
> > >> +             .gl_format = GL_BGRA_EXT,
> > >> +             .gl_type = GL_UNSIGNED_BYTE,    
> > >
> > > GL info incorrect for big-endian.    
> > 
> > As above, I don't believe that to be the case.  
> 
> I think you're right. DRM formats are always explicitly little-endian
> regardless of the machine endianess. I was probably mislead by Pixman
> formats which are machine-endian.
> 
> IOW, Pixman <-> DRM format conversion depends on machine endianess.
> 
> That makes drm_output_init_pixman() in upstream master broken for
> big-endian, because the translation from GBM formats (identical to DRM
> formats, except GBM_BO formats??) to Pixman formats does not take
> machine endianess into account.
> 
> That actually raises a fun question about wl_shm formats. If Pixman
> formats are machine-endian, and wl_shm formats are like DRM formats
> little-endian, what happens with CAIRO_FORMAT_ARGB32 drawing?
> 
> https://www.cairographics.org/manual/cairo-Image-Surfaces.html#cairo-format-t
> says it's machine-endian.
> 
> I.e. Cairo on big-endian may not work with wl_shm, because the format
> is not the ones guaranteed to work in the protocol spec or
> libwayland-server.
> 
> IOW, Pixman <-> wl_shm format conversion depends on machine endianess.
> 
> Or, are the special format codes WL_SHM_FORMAT_ARGB8888 and
> WL_SHM_FORMAT_XRGB8888 machine-endian unlike everything else?
> If these were originally designed to match Cairo formats, these would
> need to be machine-endian!
> 
> > >> +     },
> > >> +     {
> > >> +             .format = DRM_FORMAT_ARGB8888,
> > >> +             .opaque_substitute = DRM_FORMAT_XRGB8888,
> > >> +             .depth = 32,
> > >> +             .bpp = 32,
> > >> +             .gl_format = GL_BGRA_EXT,
> > >> +             .gl_type = GL_UNSIGNED_BYTE,    
> > >
> > > GL info incorrect for big-endian.
> > >
> > > Well, yeah, you know.
> > >
> > > < daniels> happy to drop a #if __BYTE_ORDER__ == BIG_ENDIAN #error
> > >            reverse literally every single one of these #endif in there
> > >
> > > Please do. :-)    
> > 
> > I went with a different approach; see below. (Also note that
> > gl-renderer is absolutely broken _today_ if this is the case, since it
> > maps ARGB8888 -> GL_BGRA_EXT + GL_UNSIGNED_BYTE without regard to
> > endianness. But I don't think it is, as above.)  
> 
> Yeah, seeing there actually are people using big-endian still
> ( https://bugs.freedesktop.org/show_bug.cgi?id=99638 ).
> 
> I've always believed gl-renderer to be broken in that exact place you
> mention, but now we have the question of what endianess was used to
> define the first wl_shm formats.


Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20170317/c414fab6/attachment.sig>