[Mesa-dev] [PATCH] mesa: Fix texture compression on big-endian systems

[Oded Gabbay]

On Tue, Sep 15, 2015 at 4:23 PM, Ulrich Weigand <uweigand at de.ibm.com> wrote:
>
> Various pieces of code to create compressed textures will first
> generate an uncompressed RGBA texture into a temporary buffer,
> and then read from that buffer while creating the final compressed
> texture in the requested format.
>
> The code reading from the temporary buffer assumes the buffer is
> formatted as an array of bytes in RGBA order.  However, the buffer
> is filled using a _mesa_texstore call with MESA_FORMAT_R8G8B8A8_UNORM
> format -- this is defined as an array of *integers* holding the
> RGBA values in packed format (least-significant to most-significant).
> This means incorrect bytes are accessed on big-endian systems.
>
> This patch fixes this by using the MESA_FORMAT_A8B8G8R8_UNORM format
> instead on big-endian systems when filling the buffer.  This fixes
> about 100 piglit test case failures on s390x for me.
>
> Signed-off-by: Ulrich Weigand <ulrich.weigand at de.ibm.com>
> ---
>  src/mesa/main/texcompress_bptc.c |    3 ++-
>  src/mesa/main/texcompress_fxt1.c |    3 ++-
>  src/mesa/main/texcompress_rgtc.c |    6 ++++--
>  src/mesa/main/texcompress_s3tc.c |    9 ++++++---
>  4 files changed, 14 insertions(+), 7 deletions(-)
>
> diff --git a/src/mesa/main/texcompress_bptc.c b/src/mesa/main/texcompress_bptc.c
> index a600180..f0f6553 100644
> --- a/src/mesa/main/texcompress_bptc.c
> +++ b/src/mesa/main/texcompress_bptc.c
> @@ -1291,7 +1291,8 @@ _mesa_texstore_bptc_rgba_unorm(TEXSTORE_PARAMS)
>        tempImageSlices[0] = (GLubyte *) tempImage;
>        _mesa_texstore(ctx, dims,
>                       baseInternalFormat,
> -                     MESA_FORMAT_R8G8B8A8_UNORM,
> +                     _mesa_little_endian() ? MESA_FORMAT_R8G8B8A8_UNORM
> +                                           : MESA_FORMAT_A8B8G8R8_UNORM,
>                       rgbaRowStride, tempImageSlices,
>                       srcWidth, srcHeight, srcDepth,
>                       srcFormat, srcType, srcAddr,
> diff --git a/src/mesa/main/texcompress_fxt1.c b/src/mesa/main/texcompress_fxt1.c
> index d605e25..ae339e1 100644
> --- a/src/mesa/main/texcompress_fxt1.c
> +++ b/src/mesa/main/texcompress_fxt1.c
> @@ -130,7 +130,8 @@ _mesa_texstore_rgba_fxt1(TEXSTORE_PARAMS)
>        tempImageSlices[0] = (GLubyte *) tempImage;
>        _mesa_texstore(ctx, dims,
>                       baseInternalFormat,
> -                     MESA_FORMAT_R8G8B8A8_UNORM,
> +                     _mesa_little_endian() ? MESA_FORMAT_R8G8B8A8_UNORM
> +                                           : MESA_FORMAT_A8B8G8R8_UNORM,
>                       rgbaRowStride, tempImageSlices,
>                       srcWidth, srcHeight, srcDepth,
>                       srcFormat, srcType, srcAddr,
> diff --git a/src/mesa/main/texcompress_rgtc.c b/src/mesa/main/texcompress_rgtc.c
> index 66de1f1..8cab7a5 100644
> --- a/src/mesa/main/texcompress_rgtc.c
> +++ b/src/mesa/main/texcompress_rgtc.c
> @@ -196,9 +196,11 @@ _mesa_texstore_rg_rgtc2(TEXSTORE_PARAMS)
>            dstFormat == MESA_FORMAT_LA_LATC2_UNORM);
>
>     if (baseInternalFormat == GL_RG)
> -      tempFormat = MESA_FORMAT_R8G8_UNORM;
> +      tempFormat = _mesa_little_endian() ? MESA_FORMAT_R8G8_UNORM
> +                                         : MESA_FORMAT_G8R8_UNORM;
>     else
> -      tempFormat = MESA_FORMAT_L8A8_UNORM;
> +      tempFormat = _mesa_little_endian() ? MESA_FORMAT_L8A8_UNORM
> +                                         : MESA_FORMAT_A8L8_UNORM;
>
>     rgRowStride = 2 * srcWidth * sizeof(GLubyte);
>     tempImage = malloc(srcWidth * srcHeight * 2 * sizeof(GLubyte));
> diff --git a/src/mesa/main/texcompress_s3tc.c b/src/mesa/main/texcompress_s3tc.c
> index 6cfe06a..7ddb0ed 100644
> --- a/src/mesa/main/texcompress_s3tc.c
> +++ b/src/mesa/main/texcompress_s3tc.c
> @@ -198,7 +198,8 @@ _mesa_texstore_rgba_dxt1(TEXSTORE_PARAMS)
>        tempImageSlices[0] = (GLubyte *) tempImage;
>        _mesa_texstore(ctx, dims,
>                       baseInternalFormat,
> -                     MESA_FORMAT_R8G8B8A8_UNORM,
> +                     _mesa_little_endian() ? MESA_FORMAT_R8G8B8A8_UNORM
> +                                           : MESA_FORMAT_A8B8G8R8_UNORM,
>                       rgbaRowStride, tempImageSlices,
>                       srcWidth, srcHeight, srcDepth,
>                       srcFormat, srcType, srcAddr,
> @@ -255,7 +256,8 @@ _mesa_texstore_rgba_dxt3(TEXSTORE_PARAMS)
>        tempImageSlices[0] = (GLubyte *) tempImage;
>        _mesa_texstore(ctx, dims,
>                       baseInternalFormat,
> -                     MESA_FORMAT_R8G8B8A8_UNORM,
> +                     _mesa_little_endian() ? MESA_FORMAT_R8G8B8A8_UNORM
> +                                           : MESA_FORMAT_A8B8G8R8_UNORM,
>                       rgbaRowStride, tempImageSlices,
>                       srcWidth, srcHeight, srcDepth,
>                       srcFormat, srcType, srcAddr,
> @@ -311,7 +313,8 @@ _mesa_texstore_rgba_dxt5(TEXSTORE_PARAMS)
>        tempImageSlices[0] = (GLubyte *) tempImage;
>        _mesa_texstore(ctx, dims,
>                       baseInternalFormat,
> -                     MESA_FORMAT_R8G8B8A8_UNORM,
> +                     _mesa_little_endian() ? MESA_FORMAT_R8G8B8A8_UNORM
> +                                           : MESA_FORMAT_A8B8G8R8_UNORM,
>                       rgbaRowStride, tempImageSlices,
>                       srcWidth, srcHeight, srcDepth,
>                       srcFormat, srcType, srcAddr,
> --
> 1.7.1
>
> --
>   Dr. Ulrich Weigand
>   GNU/Linux compilers and toolchain
>   Ulrich.Weigand at de.ibm.com
>
> _______________________________________________
> mesa-dev mailing list
> mesa-dev at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/mesa-dev

Tested it on POWER7 ppc64 and saw about the same results as Ulrich.
Checked that it didn't cause regressions on POWER8 ppc64le.

Therefore, you can add my:
Tested-by: Oded Gabbay <oded.gabbay at gmail.com>

        Oded