wl_shm formats on big-endian

[Pekka Paalanen]

On Fri, 17 Mar 2017 17:42:07 +0200
Pekka Paalanen <ppaalanen at gmail.com> wrote:

> Hi,
> 
> this is a re-send of a part of "Re: [PATCH weston 01/68] libweston: Add
> pixel-format helpers" which I believe deserves its own thread.
> 
> The core question is, what do WL_SHM_FORMAT_ARGB888 and
> WL_SHM_FORMAT_XRGB888 formats mean on big-endian?
> 
> There is a possible clash with Cairo.
> 
> As summary, format definitions are written as:
> 
> Cairo formats: machine-endian
> Pixman formats: machine-endian
> DRM formats: little-endian
> GBM formats: little-endian... BO formats?
> GL formats: machine-endian or sequence of bytes
> wl_shm formats: ???(*)
> 
> 
> *: Presumably all wl_shm formats copied from DRM formats are
> little-endian.
> 
> See below for details.

Hi all,

I have filed this issue as:
https://phabricator.freedesktop.org/T7745

The recent twist is that now there is doubt of the endianess of DRM
format codes, too.


Thanks,
pq


> On Tue, 14 Feb 2017 12:30:25 +0200
> Pekka Paalanen <ppaalanen at gmail.com> wrote:
> 
> > On Mon, 13 Feb 2017 18:24:58 +0000
> > Daniel Stone <daniel at fooishbar.org> wrote:
> >   
> 
> > > >> +     {
> > > >> +             .format = DRM_FORMAT_XRGB8888,
> > > >> +             .depth = 24,
> > > >> +             .bpp = 32,
> > > >> +             .gl_format = GL_BGRA_EXT,
> > > >> +             .gl_type = GL_UNSIGNED_BYTE,      
> > > >
> > > > GL info incorrect for big-endian.      
> > > 
> > > As above, I don't believe that to be the case.    
> > 
> > I think you're right. DRM formats are always explicitly little-endian
> > regardless of the machine endianess. I was probably mislead by Pixman
> > formats which are machine-endian.
> > 
> > IOW, Pixman <-> DRM format conversion depends on machine endianess.
> > 
> > That makes drm_output_init_pixman() in upstream master broken for
> > big-endian, because the translation from GBM formats (identical to DRM
> > formats, except GBM_BO formats??) to Pixman formats does not take
> > machine endianess into account.
> > 
> > That actually raises a fun question about wl_shm formats. If Pixman
> > formats are machine-endian, and wl_shm formats are like DRM formats
> > little-endian, what happens with CAIRO_FORMAT_ARGB32 drawing?
> > 
> > https://www.cairographics.org/manual/cairo-Image-Surfaces.html#cairo-format-t
> > says it's machine-endian.
> > 
> > I.e. Cairo on big-endian may not work with wl_shm, because the format
> > is not the ones guaranteed to work in the protocol spec or
> > libwayland-server.
> > 
> > IOW, Pixman <-> wl_shm format conversion depends on machine endianess.
> > 
> > Or, are the special format codes WL_SHM_FORMAT_ARGB8888 and
> > WL_SHM_FORMAT_XRGB8888 machine-endian unlike everything else?
> > If these were originally designed to match Cairo formats, these would
> > need to be machine-endian!
> >   
> > > >> +     },
> > > >> +     {
> > > >> +             .format = DRM_FORMAT_ARGB8888,
> > > >> +             .opaque_substitute = DRM_FORMAT_XRGB8888,
> > > >> +             .depth = 32,
> > > >> +             .bpp = 32,
> > > >> +             .gl_format = GL_BGRA_EXT,
> > > >> +             .gl_type = GL_UNSIGNED_BYTE,      
> > > >
> > > > GL info incorrect for big-endian.
> > > >
> > > > Well, yeah, you know.
> > > >
> > > > < daniels> happy to drop a #if __BYTE_ORDER__ == BIG_ENDIAN #error
> > > >            reverse literally every single one of these #endif in there
> > > >
> > > > Please do. :-)      
> > > 
> > > I went with a different approach; see below. (Also note that
> > > gl-renderer is absolutely broken _today_ if this is the case, since it
> > > maps ARGB8888 -> GL_BGRA_EXT + GL_UNSIGNED_BYTE without regard to
> > > endianness. But I don't think it is, as above.)    
> > 
> > Yeah, seeing there actually are people using big-endian still
> > ( https://bugs.freedesktop.org/show_bug.cgi?id=99638 ).
> > 
> > I've always believed gl-renderer to be broken in that exact place you
> > mention, but now we have the question of what endianess was used to
> > define the first wl_shm formats.  
> 
> 
> Thanks,
> pq

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20170419/fea305ae/attachment.sig>