[Mesa-dev] [PATCH] mesa: Fix return type of _mesa_get_format_bytes() (#37351)

[Jose Fonseca]

----- Original Message -----
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 05/19/2011 08:14 PM, Adam Jackson wrote:
> > Despite that negative values aren't sensible here, making this
> > unsigned
> > is dangerous.  Consider get_pointer_generic, which computes a value
> > of
> > the form:
> > 
> >     void *base + (int x * int stride + int y) * unsigned bpp
> > 
> > The usual arithmetic conversions will coerce the (x*stride + y)
> > subexpression to unsigned.  Since stride can be negative, this is
> > disastrous.
> > 
> > Fixes at least the following piglit tests on Ironlake:
> > 
> >     fbo/fbo-blit-d24s8
> >     spec/ARB_depth_texture/fbo-clear-formats
> >     spec/EXT_packed_depth_stencil/fbo-clear-formats
> > 
> > Signed-off-by: Adam Jackson <ajax at redhat.com>
> 
> I don't care which approach we go with.  This will likely fix it
> everywhere, but I'd like there to be a comment explaining why the
> return
> type must not be unsigned.  Otherwise someone will come along in the
> future and make it unsigned in the name of "code cleanup."

I agree with Ian: Adam's patch is good, but a comment would be useful, as the implicit coercion to unsigned is both unfortunate and unintuitive.

Jose