[Spice-devel] [spice-gtk 1/5] cursor: Avoid potential sign extension issue

[Christophe Fergeau]

When doing arithmetic operations on the uint16_t cursor width and height
with integer constants, the result of the operation will be of type 'int'
as the integer constant as type 'int'.
There are 2 places which assign the result of such an operation to
an (unsigned 64 bit)) size_t variable. This means that if width/height are
big enough, the int -> size_t conversion would cause a sign extension to
happen, which is unwanted as we are only manipulating positive values.

This commit explicitly mark the constants with the correct unsigned type.
This fixes this kind of coverity warnings:

spice-gtk-0.20/spice-gtk-0.20/gtk/channel-cursor.c:388: sign_extension:
Suspicious implicit sign extension: "hdr->height" with type "unsigned
short" (16 bits, unsigned) is promoted in "4 * hdr->width * hdr->height" to
type "int" (32 bits, signed), then sign-extended to type "unsigned long"
(64 bits, unsigned).  If "4 * hdr->width * hdr->height" is greater than
0x7FFFFFFF, the upper bits of the result will all be 1.
---
 gtk/channel-cursor.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gtk/channel-cursor.c b/gtk/channel-cursor.c
index e056b30..d33b90a 100644
--- a/gtk/channel-cursor.c
+++ b/gtk/channel-cursor.c
@@ -366,7 +366,7 @@ static display_cursor *set_cursor(SpiceChannel *channel, SpiceCursor *scursor)
 
     g_return_val_if_fail(scursor->data_size != 0, NULL);
 
-    size = 4 * hdr->width * hdr->height;
+    size = 4u * hdr->width * hdr->height;
     cursor = spice_malloc(sizeof(*cursor) + size);
     cursor->hdr = *hdr;
     cursor->default_cursor = FALSE;
@@ -404,7 +404,7 @@ static display_cursor *set_cursor(SpiceChannel *channel, SpiceCursor *scursor)
         }
         break;
     case SPICE_CURSOR_TYPE_COLOR4:
-        size = (SPICE_ALIGN(hdr->width, 2) / 2) * hdr->height;
+        size = ((unsigned int)(SPICE_ALIGN(hdr->width, 2) / 2)) * hdr->height;
         for (i = 0; i < hdr->width * hdr->height; i++) {
             pix_mask = get_pix_mask(data, size + (sizeof(uint32_t) << 4), i);
             int idx = (i & 1) ? (data[i >> 1] & 0x0f) : ((data[i >> 1] & 0xf0) >> 4);
-- 
1.8.4.2