[Mesa-dev] [PATCH] mesa: Fix return type of _mesa_get_format_bytes() (#37351)

[Chad Versace]

On 05/19/2011 11:14 AM, Adam Jackson wrote:
> Despite that negative values aren't sensible here, making this unsigned
> is dangerous.  Consider get_pointer_generic, which computes a value of
> the form:
> 
>     void *base + (int x * int stride + int y) * unsigned bpp
> 
> The usual arithmetic conversions will coerce the (x*stride + y)
> subexpression to unsigned.  Since stride can be negative, this is
> disastrous.
> 
> Fixes at least the following piglit tests on Ironlake:
> 
>     fbo/fbo-blit-d24s8
>     spec/ARB_depth_texture/fbo-clear-formats
>     spec/EXT_packed_depth_stencil/fbo-clear-formats
> 
> Signed-off-by: Adam Jackson <ajax at redhat.com>
> ---
>  src/mesa/main/formats.c |    2 +-
>  src/mesa/main/formats.h |    2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)


I was just about to submit my own patch for this bug, but I you and idr
were too quick. My patch resembled idr's, but I prefer your approach. It
doesn't rely on users of _mesa_get_format_bytes() to remember to do an
explicit cast.

Reviewed-by: Chad Versace <chad.versace at intel.com>


-- 
Chad Versace
chad at chad-versace.us