VNC patch for Xserver 1.3

[Stefan Dirsch]

On Thu, Aug 23, 2007 at 04:31:02PM +0200, Stefan Dirsch wrote:
> Thanks. Unfortunately the Xserver crashes immediately now.
> 
> (gdb) run
> 
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x2ab337bb4290 (LWP 4737)]
> 0x00000000004a0bdc in xf86CollectInputOptions ()
> (gdb) bt
> #0  0x00000000004a0bdc in xf86CollectInputOptions ()
> #1  0x00002ab338d5a019 in xf86rfbKeybInit ()
>    from /usr/lib64/xorg/modules//extensions/libvnc.so
> #2  0x0000000000467fdc in InitInput ()
> #3  0x0000000000439931 in main ()
> (gdb) quit

Tried again with X.Org 7.3 and current xf4vnc CVS. Still crashing. :-)
Here are the results of my investigations.

Backtrace:
0: X(xf86SigHandler+0x6a) [0x48ce3a]
1: /lib64/libc.so.6 [0x2b2b2e271ba0]
2: /lib64/libc.so.6(strlen+0x40) [0x2b2b2e2b6ba0]
3: X(xf86ActivateDevice+0x59) [0x47b389]
4: X(InitInput+0x4d) [0x46785d]
5: X(main+0x370) [0x436210]
6: /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b2b2e25eb24]
7: X(FontFileCompleteXLFD+0x259) [0x435659]

Program received signal SIGSEGV, Segmentation fault.
0x00002b587f48eba0 in strlen () from /lib64/libc.so.6
(gdb) bt
#0  0x00002b587f48eba0 in strlen () from /lib64/libc.so.6
#1  0x000000000047b389 in xf86ActivateDevice (local=0x8be9d0)
    at xf86Xinput.c:158
#2  0x000000000046785d in InitInput (argc=<value optimized out>, 
    argv=<value optimized out>) at xf86Init.c:960
#3  0x0000000000436210 in main (argc=1, argv=0x7fff2ce5c1a8, envp=0x83a090)
    at main.c:397

158             local->atom = MakeAtom(local->type_name,
159                                    strlen(local->type_name),
160                                    TRUE);

(gdb) p* local
$3 = {next = 0x8be900, name = 0x7ef4d0 "Mouse[3]", flags = 78, 
  device_control = 0x2b0c70993820 <xf86rfbMouseControlProc>, read_input = 0, 
  control_proc = 0, close_proc = 0, switch_mode = 0, conversion_proc = 0, 
  reverse_conversion_proc = 0, set_device_valuators = 0, fd = -1, atom = 0, 
  dev = 0x0, private = 0x0, private_flags = 0, first = 0, last = 0, old_x = 0, 
  old_y = 0, type_name = 0x0, always_core_feedback = 0x2b0c709acb3c, 
  conf_idev = 0x0, drv = 0x7f5000, module = 0x0, options = 0x0, 
  history_size = 256}

Looks like the pInfo structure gets not filled properly. I'm not sure,
where this is done. Here is the configuration I'm currently using.

Section "InputDevice"
  Driver       "rfbkeyb"
  Identifier   "Keyboard[2]"
  Option       "InputFashion" "VNC"
EndSection

Section "InputDevice"
  Driver       "rfbmouse"
  Identifier   "Mouse[3]"
  Option       "InputFashion" "VNC"
EndSection

Section "ServerLayout"
  [...]
  InputDevice  "Keyboard[2]" "ExtraKeyboard"
  InputDevice  "Mouse[3]" "ExtraPointer"
EndSection

You get different results for pInfo for a regular mouse section and of
course no crashes.

Section "InputDevice"
  Driver       "mouse"
  Identifier   "Mouse[1]"
  Option       "Buttons" "5"
  Option       "Device" "/dev/input/mice"
  Option       "Name" "MX310"
  Option       "Protocol" "ExplorerPS/2"
  Option       "Vendor" "Logitech"
  Option       "ZAxisMapping" "4 5"
EndSection

(gdb) p* local
$2 = {next = 0x8b9230, name = 0x7ee9d0 "Mouse[1]", flags = 78, 
  device_control = 0x2b61f0cebea0 <MouseProc>, 
  read_input = 0x2b61f0ceb1a0 <MouseReadInput>, control_proc = 0, 
  close_proc = 0, switch_mode = 0, 
  conversion_proc = 0x2b61f0ce86a0 <MouseConvert>, 
  reverse_conversion_proc = 0, set_device_valuators = 0, fd = -1, atom = 0, 
  dev = 0x0, private = 0x8be180, private_flags = 0, first = 0, last = 0, 
  old_x = 0, old_y = 0, type_name = 0x2b61f0cedd55 "MOUSE", 
  always_core_feedback = 0x0, conf_idev = 0x7f4e90, drv = 0x808070, 
  module = 0x8be0a0, options = 0x8bda80, history_size = 256}

Alan, can you at least reproduce this problem?

Best regards,
Stefan

Public Key available
------------------------------------------------------
Stefan Dirsch (Res. & Dev.)   SUSE LINUX Products GmbH
Tel: 0911-740 53 0            Maxfeldstraße 5
FAX: 0911-740 53 479          D-90409 Nürnberg
http://www.suse.de            Germany 
-----------------------------------------------------------------
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
-----------------------------------------------------------------