[Mesa-dev] [PATCH] mesa: Fix return type of _mesa_get_format_bytes() (#37351)

Adam Jackson ajax at redhat.com
Thu May 19 11:14:14 PDT 2011

Despite that negative values aren't sensible here, making this unsigned
is dangerous.  Consider get_pointer_generic, which computes a value of
the form:

    void *base + (int x * int stride + int y) * unsigned bpp

The usual arithmetic conversions will coerce the (x*stride + y)
subexpression to unsigned.  Since stride can be negative, this is

Fixes at least the following piglit tests on Ironlake:


Signed-off-by: Adam Jackson <ajax at redhat.com>
 src/mesa/main/formats.c |    2 +-
 src/mesa/main/formats.h |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/mesa/main/formats.c b/src/mesa/main/formats.c
index 60e8ae3..a46633b 100644
--- a/src/mesa/main/formats.c
+++ b/src/mesa/main/formats.c
@@ -1119,7 +1119,7 @@ _mesa_get_format_name(gl_format format)
  * Normally, a block is 1x1 (a single pixel).  But for compressed formats
  * a block may be 4x4 or 8x4, etc.
 _mesa_get_format_bytes(gl_format format)
    const struct gl_format_info *info = _mesa_get_format_info(format);
diff --git a/src/mesa/main/formats.h b/src/mesa/main/formats.h
index df9ed70..0640bbc 100644
--- a/src/mesa/main/formats.h
+++ b/src/mesa/main/formats.h
@@ -216,7 +216,7 @@ typedef enum
 extern const char *
 _mesa_get_format_name(gl_format format);
-extern GLuint
+extern GLint
 _mesa_get_format_bytes(gl_format format);
 extern GLint

More information about the mesa-dev mailing list