Xserver Compatibility with X11R4 Xt Maximum Keycode Bug

[John Bower]

Hello,

Perhaps the following information will be helpful to anyone wishing to
display legacy Xt-based applications to modern X server  
implementations. This
includes applications built with SUN's OPEN LOOK Intrinsics Toolkit  
(OLIT).

Some time ago, a bug existed in X11R4 X Toolkit Intrinsics (Xt) within
XtKeysymToKeycodeList() that caused Xt-based X clients to crash with
segmentation fault. There are references explaining this in more detail
in a few February 1994 posts to comp.unix.aix and comp.sys.sgi.bugs:

http://groups.google.com/group/comp.unix.aix/browse_thread/thread/ 
a02e22f06b244f0c

Very briefly, X clients built with X11R4 Xt can crash when calling
XtKeysymToKeycodeList() if the X server indicates maximum keycode value
of 255 and the keysym being searched for is not found.

The problem was fixed in X11R5 X Toolkit Intrinsics, but legacy
applications which have the bug still exist, of course :-) . If they
can't be fixed (e.g. no access to source code), the problem can be
avoided by ensuring all sought-after keysyms for an application have
been mapped.

Here's an example of this involving SUN's OPEN LOOK Intrinsics Toolkit
(OLIT), which uses Xt:

http://groups.google.com/group/comp.unix.solaris/browse_thread/thread/ 
baf79f4f692bc673

Another approach is to configure the X server to report a maximum
keycode value of 254, as indicated in the Feb 1994 usenet posts to
the groups comp.unix.aix and comp.sys.sgi.bugs.

If you're using the X Keyboard Extension, this can be done in the
extension's "keycodes" configuration files. For example, here's such a
change for my installation of FedoraCore4-based YellowDog Linux 4.1,
within file /etc/X11/xkb/keycodes/xfree86, in the definition of
xkb_keycodes "basic" (ellipses indicate lines omitted for brevity):

     xkb_keycodes "basic" {

         minimum= 8;
         maximum= 254;
         ...

     }

If you can recompile your X server, you might change a macro and
recompile. For example, in X11R6.9.0, within file
xc/programs/Xserver/hw/xfree86/common/atKeynames.h, I can define
NUM_KEYCODES to be 247 instead of 248, which fixes the problem for my
particular setup when I'm using the core keyboard mechanism (not
X Keyboard Extension):

     #define GLYPHS_PER_KEY  4
     #define NUM_KEYCODES    247
     #define MIN_KEYCODE     8
     #define MAX_KEYCODE     (NUM_KEYCODES + MIN_KEYCODE - 1)

Another option is to change a function in the server. For example, in
X11R6.9.0, within file xc/programs/Xserver/dix/devices.c, I can define
QueryMinMaxKeyCodes() to report 254 for maximum keycode whenever it's
255:

     void
     QueryMinMaxKeyCodes(KeyCode *minCode, KeyCode *maxCode)
     {
         if (inputInfo.keyboard) {
             *minCode = inputInfo.keyboard->key->curKeySyms.minKeyCode;
             *maxCode = inputInfo.keyboard->key->curKeySyms.maxKeyCode;
             if (*maxCode == 255) *maxCode = 254;
         }
     }

I welcome any other information, suggestions, or comments regarding this
issue.

-john

John Bower
jbower at columbus.rr.com
Columbus, OH