[Pixman] [PATCH] In pixman_image_create_bits() allow images larger than 2GB

[Soeren Sandmann]

Chris Wilson <chris at chris-wilson.co.uk> writes:

> On Thu, 11 Aug 2011 06:43:03 -0400, Søren Sandmann <sandmann at cs.au.dk> wrote:
>> From: Søren Sandmann Pedersen <ssp at redhat.com>
>> 
>> There is no reason for pixman_image_create_bits() to check that the
>> image size fits in int32_t. The correct check is against size_t since
>> that is what the argument to calloc() is.
>
> The only question I have is whether ssize_t width and height is
> appropriate, the valid image dimensions are still only int32_t right?

Yeah, you are right. It's only an internal function, so using ssize_t is
harmless, but there's no need to introduce this confusion. New version
below.


Thanks,
Soren



commit cb49f9427ead511b9bd60e771fc1b0d74d72d22a
Author: Søren Sandmann Pedersen <ssp at redhat.com>
Date:   Thu Aug 11 06:30:43 2011 -0400

    In pixman_image_create_bits() allow images larger than 2GB
    
    There is no reason for pixman_image_create_bits() to check that the
    image size fits in int32_t. The correct check is against size_t since
    that is what the argument to calloc() is.
    
    This patch fixes this by adding a new _pixman_multiply_overflows_size()
    and using it in create_bits(). Also prepend an underscore to the names
    of other similar functions since they are internal to pixman.
    
    V2: Use int, not ssize_t for the arguments in create_bits() since
    width/height are still limited to 32 bits, as pointed out by Chris
    Wilson.

diff --git a/pixman/pixman-bits-image.c b/pixman/pixman-bits-image.c
index 4e9ed14..f5b66dc 100644
--- a/pixman/pixman-bits-image.c
+++ b/pixman/pixman-bits-image.c
@@ -1490,10 +1490,10 @@ static uint32_t *
 create_bits (pixman_format_code_t format,
              int                  width,
              int                  height,
-             int *                rowstride_bytes)
+             int *		  rowstride_bytes)
 {
     int stride;
-    int buf_size;
+    size_t buf_size;
     int bpp;
 
     /* what follows is a long-winded way, avoiding any possibility of integer
@@ -1502,11 +1502,11 @@ create_bits (pixman_format_code_t format,
      */
 
     bpp = PIXMAN_FORMAT_BPP (format);
-    if (pixman_multiply_overflows_int (width, bpp))
+    if (_pixman_multiply_overflows_int (width, bpp))
 	return NULL;
 
     stride = width * bpp;
-    if (pixman_addition_overflows_int (stride, 0x1f))
+    if (_pixman_addition_overflows_int (stride, 0x1f))
 	return NULL;
 
     stride += 0x1f;
@@ -1514,7 +1514,7 @@ create_bits (pixman_format_code_t format,
 
     stride *= sizeof (uint32_t);
 
-    if (pixman_multiply_overflows_int (height, stride))
+    if (_pixman_multiply_overflows_size (height, stride))
 	return NULL;
 
     buf_size = height * stride;
diff --git a/pixman/pixman-private.h b/pixman/pixman-private.h
index 6a3935e..a25897d 100644
--- a/pixman/pixman-private.h
+++ b/pixman/pixman-private.h
@@ -691,10 +691,13 @@ void *
 pixman_malloc_abc (unsigned int a, unsigned int b, unsigned int c);
 
 pixman_bool_t
-pixman_multiply_overflows_int (unsigned int a, unsigned int b);
+_pixman_multiply_overflows_size (size_t a, size_t b);
 
 pixman_bool_t
-pixman_addition_overflows_int (unsigned int a, unsigned int b);
+_pixman_multiply_overflows_int (unsigned int a, unsigned int b);
+
+pixman_bool_t
+_pixman_addition_overflows_int (unsigned int a, unsigned int b);
 
 /* Compositing utilities */
 void
diff --git a/pixman/pixman-utils.c b/pixman/pixman-utils.c
index cb4e621..49e3488 100644
--- a/pixman/pixman-utils.c
+++ b/pixman/pixman-utils.c
@@ -31,15 +31,19 @@
 #include "pixman-private.h"
 
 pixman_bool_t
-pixman_multiply_overflows_int (unsigned int a,
-                               unsigned int b)
+_pixman_multiply_overflows_size (size_t a, size_t b)
+{
+    return a >= SIZE_MAX / b;
+}
+
+pixman_bool_t
+_pixman_multiply_overflows_int (unsigned int a, unsigned int b)
 {
     return a >= INT32_MAX / b;
 }
 
 pixman_bool_t
-pixman_addition_overflows_int (unsigned int a,
-                               unsigned int b)
+_pixman_addition_overflows_int (unsigned int a, unsigned int b)
 {
     return a > INT32_MAX - b;
 }