[Mesa-dev] [PATCH] llvmpipe: fix overflow bug in total texture size computation

[Brian Paul]

Add size_t casts when multiplying slice size by number of slices to
avoid 32-bit uint overflow.  This bug has been here for a long time.
But before the recent proxy texture changes, core Mesa was detecting
that the texture was too large and we never got this far.

Fixes https://bugs.freedesktop.org/show_bug.cgi?id=55117
---
 src/gallium/drivers/llvmpipe/lp_texture.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/src/gallium/drivers/llvmpipe/lp_texture.c b/src/gallium/drivers/llvmpipe/lp_texture.c
index c0a612c..0aa1299 100644
--- a/src/gallium/drivers/llvmpipe/lp_texture.c
+++ b/src/gallium/drivers/llvmpipe/lp_texture.c
@@ -172,7 +172,9 @@ llvmpipe_texture_layout(struct llvmpipe_screen *screen,
          }
       }
 
-      total_size += lpr->num_slices_faces[level] * lpr->img_stride[level];
+      total_size += (size_t) lpr->num_slices_faces[level]
+                  * (size_t) lpr->img_stride[level];
+
       if (total_size > LP_MAX_TEXTURE_SIZE) {
          goto fail;
       }
-- 
1.7.3.4