[Mesa-dev] [PATCH] llvmpipe: fix overflow bug in total texture size computation

[Roland Scheidegger]

Good catch.
I'm not sure though using size_t casts (and size_t sized total_value)
is good enough since this could also be hit on archs using 32bit size_t?
Though i guess using 64bit arithmetic on 32bit would be sort of slow...

Roland


Am 19.09.2012 21:30, schrieb Brian Paul:
> Add size_t casts when multiplying slice size by number of slices to
> avoid 32-bit uint overflow.  This bug has been here for a long time.
> But before the recent proxy texture changes, core Mesa was detecting
> that the texture was too large and we never got this far.
> 
> Fixes https://bugs.freedesktop.org/show_bug.cgi?id=55117
> ---
>  src/gallium/drivers/llvmpipe/lp_texture.c |    4 +++-
>  1 files changed, 3 insertions(+), 1 deletions(-)
> 
> diff --git a/src/gallium/drivers/llvmpipe/lp_texture.c b/src/gallium/drivers/llvmpipe/lp_texture.c
> index c0a612c..0aa1299 100644
> --- a/src/gallium/drivers/llvmpipe/lp_texture.c
> +++ b/src/gallium/drivers/llvmpipe/lp_texture.c
> @@ -172,7 +172,9 @@ llvmpipe_texture_layout(struct llvmpipe_screen *screen,
>           }
>        }
>  
> -      total_size += lpr->num_slices_faces[level] * lpr->img_stride[level];
> +      total_size += (size_t) lpr->num_slices_faces[level]
> +                  * (size_t) lpr->img_stride[level];
> +
>        if (total_size > LP_MAX_TEXTURE_SIZE) {
>           goto fail;
>        }
>