[Mesa-dev] [PATCH] mesa: Fix return type of _mesa_get_format_bytes() (#37351)

[Ian Romanick]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/19/2011 08:14 PM, Adam Jackson wrote:
> Despite that negative values aren't sensible here, making this unsigned
> is dangerous.  Consider get_pointer_generic, which computes a value of
> the form:
> 
>     void *base + (int x * int stride + int y) * unsigned bpp
> 
> The usual arithmetic conversions will coerce the (x*stride + y)
> subexpression to unsigned.  Since stride can be negative, this is
> disastrous.
> 
> Fixes at least the following piglit tests on Ironlake:
> 
>     fbo/fbo-blit-d24s8
>     spec/ARB_depth_texture/fbo-clear-formats
>     spec/EXT_packed_depth_stencil/fbo-clear-formats
> 
> Signed-off-by: Adam Jackson <ajax at redhat.com>

I don't care which approach we go with.  This will likely fix it
everywhere, but I'd like there to be a comment explaining why the return
type must not be unsigned.  Otherwise someone will come along in the
future and make it unsigned in the name of "code cleanup."

> ---
>  src/mesa/main/formats.c |    2 +-
>  src/mesa/main/formats.h |    2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/src/mesa/main/formats.c b/src/mesa/main/formats.c
> index 60e8ae3..a46633b 100644
> --- a/src/mesa/main/formats.c
> +++ b/src/mesa/main/formats.c
> @@ -1119,7 +1119,7 @@ _mesa_get_format_name(gl_format format)
>   * Normally, a block is 1x1 (a single pixel).  But for compressed formats
>   * a block may be 4x4 or 8x4, etc.
>   */
> -GLuint
> +GLint
>  _mesa_get_format_bytes(gl_format format)
>  {
>     const struct gl_format_info *info = _mesa_get_format_info(format);
> diff --git a/src/mesa/main/formats.h b/src/mesa/main/formats.h
> index df9ed70..0640bbc 100644
> --- a/src/mesa/main/formats.h
> +++ b/src/mesa/main/formats.h
> @@ -216,7 +216,7 @@ typedef enum
>  extern const char *
>  _mesa_get_format_name(gl_format format);
>  
> -extern GLuint
> +extern GLint
>  _mesa_get_format_bytes(gl_format format);
>  
>  extern GLint
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAk3WB0YACgkQX1gOwKyEAw/VVwCeM2EGDlT7pf0zImg++KBQMQ0Y
8zEAniTakBqiEAj0mxCFNypyJ86EhGyl
=NP7k
-----END PGP SIGNATURE-----