[Glamor] [PATCH 14/34] glamor: Fix some integer overflow errors.

[Alex Deucher]

From: Eric Anholt <eric at anholt.net>

Imagine a nbox that was (UINT_MAX + small number) / 4 * 2 *
sizeof(float). We'd malloc a few bytes after the integer overflow,
but glamor_set_normalize_vcoords would write over gigabytes of heap.

Ported from Eric's xserver glamor tree.

Signed-off-by: Eric Anholt <eric at anholt.net>
Signed-off-by: Alex Deucher <alexander.deucher at amd.com>
---
 src/glamor_fill.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/glamor_fill.c b/src/glamor_fill.c
index 4eb23cd..634dbe1 100644
--- a/src/glamor_fill.c
+++ b/src/glamor_fill.c
@@ -204,10 +204,10 @@ _glamor_solid_boxes(PixmapPtr pixmap, BoxPtr box, int nbox, float *color)
 
 	pixmap_priv_get_dest_scale(pixmap_priv, &xscale, &yscale);
 
-	if (nbox * 4 * 2 > ARRAY_SIZE(vertices)) {
+	if (nbox > valid_nbox) {
 		int allocated_box;
 
-		if (nbox * 6 > GLAMOR_COMPOSITE_VBO_VERT_CNT) {
+		if (nbox > GLAMOR_COMPOSITE_VBO_VERT_CNT / 6) {
 			allocated_box = GLAMOR_COMPOSITE_VBO_VERT_CNT / 6;
 		} else
 			allocated_box = nbox;
-- 
1.8.3.1