[PATCH] Return Success from generate_modkeymap() when max_keys_per_mod is zero

[Peter Hutterer]

On Wed, Aug 25, 2010 at 10:38:40AM +0200, Adam Tkac wrote:
> Hello,
> 
> simple patch which corrects dix/inpututils.c:generate_modkeymap()
> function is attached. Current generate_modkeymap function is broken
> and returns BadAlloc when keyboard has no modifiers (i.e.
> max_keys_per_mod is zero).
> 
> Although this bug has no effect for Xorg server ifself, it has effect
> for Xvnc VNC server.
> 
> I've also checked parts of Xorg source which might be affected by this
> change (because returned modkeymap can be NULL with the patch) and
> this change doesn't cause any problem.
> 
> Patch applies to server-1.9-branch but apply it also to server-1.8-branch
> and server-1.7-branch branches, please.
> 
> Regards, Adam
> 
> -- 
> Adam Tkac, Red Hat, Inc.

> From 3fae47581a47613f1117b8794ae37b75ace73f3e Mon Sep 17 00:00:00 2001
> From: Adam Tkac <atkac at redhat.com>
> Date: Tue, 24 Aug 2010 17:21:30 +0200
> Subject: [PATCH] Return Success from generate_modkeymap() when max_keys_per_mod is zero.
> 
> max_keys_per_mod equal to zero is a valid situation so generate_modkeymap
> should not return BadAlloc in this case.
> 
> Signed-off-by: Adam Tkac <atkac at redhat.com>
> ---
>  dix/inpututils.c |   24 +++++++++++++-----------
>  1 files changed, 13 insertions(+), 11 deletions(-)
> 
> diff --git a/dix/inpututils.c b/dix/inpututils.c
> index 8ec80b5..6693c67 100644
> --- a/dix/inpututils.c
> +++ b/dix/inpututils.c
> @@ -286,7 +286,7 @@ int generate_modkeymap(ClientPtr client, DeviceIntPtr dev,
>  {
>      CARD8 keys_per_mod[8];
>      int max_keys_per_mod;
> -    KeyCode *modkeymap;
> +    KeyCode *modkeymap = NULL;
>      int i, j, ret;
>  
>      ret = XaceHook(XACE_DEVICE_ACCESS, client, dev, DixGetAttrAccess);
> @@ -310,18 +310,20 @@ int generate_modkeymap(ClientPtr client, DeviceIntPtr dev,
>          }
>      }
>  
> -    modkeymap = calloc(max_keys_per_mod * 8, sizeof(KeyCode));
> -    if (!modkeymap)
> -        return BadAlloc;
> +    if (max_keys_per_mod != 0) {
> +        modkeymap = calloc(max_keys_per_mod * 8, sizeof(KeyCode));
> +        if (!modkeymap)
> +            return BadAlloc;
>  
> -    for (i = 0; i < 8; i++)
> -        keys_per_mod[i] = 0;
> +        for (i = 0; i < 8; i++)
> +            keys_per_mod[i] = 0;
>  
> -    for (i = 8; i < MAP_LENGTH; i++) {
> -        for (j = 0; j < 8; j++) {
> -            if (dev->key->xkbInfo->desc->map->modmap[i] & (1 << j)) {
> -                modkeymap[(j * max_keys_per_mod) + keys_per_mod[j]] = i;
> -                keys_per_mod[j]++;
> +        for (i = 8; i < MAP_LENGTH; i++) {
> +            for (j = 0; j < 8; j++) {
> +                if (dev->key->xkbInfo->desc->map->modmap[i] & (1 << j)) {
> +                    modkeymap[(j * max_keys_per_mod) + keys_per_mod[j]] = i;
> +                    keys_per_mod[j]++;
> +                }
>              }
>          }
>      }
> -- 
> 1.7.2.2

merged into input-next, thanks. will be upstream soon.

Cheers,
  Peter