[PATCH:libX11] Fix overflow checks in _XkbReadKeySyms when key_sym_map is already created

[Alan Coopersmith]

We were checking to make sure that the largest keysym value was within
the range of the allocated buffer, but checking against different limits
in the not-yet-allocated vs. the already-allocated branches.

The check should be the same in both, and reflect the size used for the
allocation, which is based on the maximum key code value, so we move it
to be a common check, before we branch, instead of duplicating in each
branch.

map->key_sym_map is an array of XkbSymMapRec structs, [0..max_key_code]
map->syms is the array for which num_syms is recorded, hence is not the
right value to check for ensuring our key_sym_map accesses are in range.

Signed-off-by: Alan Coopersmith <alan.coopersmith at oracle.com>
Reported-by: Barry Kauler <bkauler at gmail.com>
Tested-by: Barry Kauler <bkauler at gmail.com>
---
 src/xkb/XKBGetMap.c |    9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/src/xkb/XKBGetMap.c b/src/xkb/XKBGetMap.c
index c73e655..a24ddd8 100644
--- a/src/xkb/XKBGetMap.c
+++ b/src/xkb/XKBGetMap.c
@@ -147,15 +147,16 @@ _XkbReadKeySyms(XkbReadBufferPtr buf,XkbDescPtr xkb,xkbGetMapReply *rep)
 {
 register int i;
 XkbClientMapPtr	map;
+int size = xkb->max_key_code + 1;
+
+    if (((unsigned short)rep->firstKeySym + rep->nKeySyms) > size)
+	return BadLength;
 
     map= xkb->map;
     if (map->key_sym_map==NULL) {
 	register int offset;
-	int size = xkb->max_key_code + 1;
 	XkbSymMapPtr	oldMap;
 	xkbSymMapWireDesc *newMap;
-	if (((unsigned short)rep->firstKeySym + rep->nKeySyms) > size)
-	    return BadLength;
 	map->key_sym_map= _XkbTypedCalloc(size,XkbSymMapRec);
 	if (map->key_sym_map==NULL)
 	    return BadAlloc;
@@ -212,8 +213,6 @@ XkbClientMapPtr	map;
 	KeySym *		newSyms;
 	int			tmp;
 
-	if (((unsigned short)rep->firstKeySym + rep->nKeySyms) > map->num_syms)
-	    return BadLength;
 	oldMap = &map->key_sym_map[rep->firstKeySym];
 	for (i=0;i<(int)rep->nKeySyms;i++,oldMap++) {
 	    newMap= (xkbSymMapWireDesc *)
-- 
1.7.9.2