VNC patch for Xserver 1.3

[Stefan Dirsch]

On Thu, Oct 18, 2007 at 09:21:21AM +0100, Alan Hourihane wrote:
> On Wed, 2007-10-17 at 13:08 +0200, Stefan Dirsch wrote:
> > On Thu, Aug 23, 2007 at 04:31:02PM +0200, Stefan Dirsch wrote:
> > > Thanks. Unfortunately the Xserver crashes immediately now.
> > > 
> > > (gdb) run
> > > 
> > > Program received signal SIGSEGV, Segmentation fault.
> > > [Switching to Thread 0x2ab337bb4290 (LWP 4737)]
> > > 0x00000000004a0bdc in xf86CollectInputOptions ()
> > > (gdb) bt
> > > #0  0x00000000004a0bdc in xf86CollectInputOptions ()
> > > #1  0x00002ab338d5a019 in xf86rfbKeybInit ()
> > >    from /usr/lib64/xorg/modules//extensions/libvnc.so
> > > #2  0x0000000000467fdc in InitInput ()
> > > #3  0x0000000000439931 in main ()
> > > (gdb) quit
> > 
> > Tried again with X.Org 7.3 and current xf4vnc CVS. Still crashing. :-)
> > Here are the results of my investigations.
> > 
> > Backtrace:
> > 0: X(xf86SigHandler+0x6a) [0x48ce3a]
> > 1: /lib64/libc.so.6 [0x2b2b2e271ba0]
> > 2: /lib64/libc.so.6(strlen+0x40) [0x2b2b2e2b6ba0]
> > 3: X(xf86ActivateDevice+0x59) [0x47b389]
> > 4: X(InitInput+0x4d) [0x46785d]
> > 5: X(main+0x370) [0x436210]
> > 6: /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b2b2e25eb24]
> > 7: X(FontFileCompleteXLFD+0x259) [0x435659]
> > 
> > Program received signal SIGSEGV, Segmentation fault.
> > 0x00002b587f48eba0 in strlen () from /lib64/libc.so.6
> > (gdb) bt
> > #0  0x00002b587f48eba0 in strlen () from /lib64/libc.so.6
> > #1  0x000000000047b389 in xf86ActivateDevice (local=0x8be9d0)
> >     at xf86Xinput.c:158
> > #2  0x000000000046785d in InitInput (argc=<value optimized out>, 
> >     argv=<value optimized out>) at xf86Init.c:960
> > #3  0x0000000000436210 in main (argc=1, argv=0x7fff2ce5c1a8, envp=0x83a090)
> >     at main.c:397
> > 
> > 158             local->atom = MakeAtom(local->type_name,
> > 159                                    strlen(local->type_name),
> > 160                                    TRUE);
> > 
> > (gdb) p* local
> > $3 = {next = 0x8be900, name = 0x7ef4d0 "Mouse[3]", flags = 78, 
> >   device_control = 0x2b0c70993820 <xf86rfbMouseControlProc>, read_input = 0, 
> >   control_proc = 0, close_proc = 0, switch_mode = 0, conversion_proc = 0, 
> >   reverse_conversion_proc = 0, set_device_valuators = 0, fd = -1, atom = 0, 
> >   dev = 0x0, private = 0x0, private_flags = 0, first = 0, last = 0, old_x = 0, 
> >   old_y = 0, type_name = 0x0, always_core_feedback = 0x2b0c709acb3c, 
> >   conf_idev = 0x0, drv = 0x7f5000, module = 0x0, options = 0x0, 
> >   history_size = 256}
> > 
> > Looks like the pInfo structure gets not filled properly. I'm not sure,
> > where this is done. Here is the configuration I'm currently using.
> > 
> > Section "InputDevice"
> >   Driver       "rfbkeyb"
> >   Identifier   "Keyboard[2]"
> >   Option       "InputFashion" "VNC"
> > EndSection
> > 
> > Section "InputDevice"
> >   Driver       "rfbmouse"
> >   Identifier   "Mouse[3]"
> >   Option       "InputFashion" "VNC"
> > EndSection
> > 
> > Section "ServerLayout"
> >   [...]
> >   InputDevice  "Keyboard[2]" "ExtraKeyboard"
> >   InputDevice  "Mouse[3]" "ExtraPointer"
> > EndSection
> > 
> > You get different results for pInfo for a regular mouse section and of
> > course no crashes.
> > 
> > Section "InputDevice"
> >   Driver       "mouse"
> >   Identifier   "Mouse[1]"
> >   Option       "Buttons" "5"
> >   Option       "Device" "/dev/input/mice"
> >   Option       "Name" "MX310"
> >   Option       "Protocol" "ExplorerPS/2"
> >   Option       "Vendor" "Logitech"
> >   Option       "ZAxisMapping" "4 5"
> > EndSection
> > 
> > (gdb) p* local
> > $2 = {next = 0x8b9230, name = 0x7ee9d0 "Mouse[1]", flags = 78, 
> >   device_control = 0x2b61f0cebea0 <MouseProc>, 
> >   read_input = 0x2b61f0ceb1a0 <MouseReadInput>, control_proc = 0, 
> >   close_proc = 0, switch_mode = 0, 
> >   conversion_proc = 0x2b61f0ce86a0 <MouseConvert>, 
> >   reverse_conversion_proc = 0, set_device_valuators = 0, fd = -1, atom = 0, 
> >   dev = 0x0, private = 0x8be180, private_flags = 0, first = 0, last = 0, 
> >   old_x = 0, old_y = 0, type_name = 0x2b61f0cedd55 "MOUSE", 
> >   always_core_feedback = 0x0, conf_idev = 0x7f4e90, drv = 0x808070, 
> >   module = 0x8be0a0, options = 0x8bda80, history_size = 256}
> > 
> > Alan, can you at least reproduce this problem?
> 
> I'll try and get to this asap.

Could you try this on a x86_64 system? It doesn't seem to happen on a
32bit x86 system.

Best regards,
Stefan

Public Key available
------------------------------------------------------
Stefan Dirsch (Res. & Dev.)   SUSE LINUX Products GmbH
Tel: 0911-740 53 0            Maxfeldstraße 5
FAX: 0911-740 53 479          D-90409 Nürnberg
http://www.suse.de            Germany 
-----------------------------------------------------------------
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
-----------------------------------------------------------------