Strange libXrender behaviour on Solaris10/SPARC

[Nicolai Stange]

Hi everybody,

I've got a really strange problem concerning libXrender on
Solaris10/SPARC.

In order to build the current Mozilla Firefox and in order to decouple
my self-compiled packages from Blastwave's X-stuff in /opt/csw/lib, I
decided to build the whole xlib on my own.

I took the current sources from
ftp://ftp.freedesktop.org/pub/xorg/X11R7.5/src/{lib,proto}, built the
current GTK+ on it and compiled Mozilla Firefox (it doesn't matter
whether to use the GCC or the Sun C compiler, the problem arises in
both cases).

Now, holding my breath, I entered 'firefox' in my ssh shell on that
Solaris10/SPARC box and boooom, it crashed:
###!!! ABORT: X_GrabButton: BadLength (poly request too large or
internal Xlib length error); 3193 requests ago:
file /tmp/xas/build/mozilla_firefox_3.6.9_default_gcc_32/src/mozilla-1.9.2/toolkit/xre/nsX11ErrorHandler.cpp, line 182
feca2590 /tmp/xas/build/mozilla_firefox_3.6.9_default_gcc_32/src/mozilla-1.9.2/obj-sparc-sun-solaris2.10/browser/toolkit/library/libxul.so:NS_DebugBreak_P+0x1f0
fe1bc0ac /tmp/xas/build/mozilla_firefox_3.6.9_default_gcc_32/src/mozilla-1.9.2/obj-sparc-sun-solaris2.10/browser/toolkit/library/libxul.so:_Z25NS_CreateNativeAppSupportPP19nsINativeAppSupport+0x24c
fd268e20 /pf/m/m222086/xas/solaris10/gtk/libX11-1.3.2/lib/libX11.so.6.3.0:_XError+0x190
fd276ff8 /pf/m/m222086/xas/solaris10/gtk/libX11-1.3.2/lib/libX11.so.6.3.0:_XFreeX11XCBStructure+0xaa8
fd2772c8 /pf/m/m222086/xas/solaris10/gtk/libX11-1.3.2/lib/libX11.so.6.3.0:_XEventsQueued+0x90
fd2499c4 /pf/m/m222086/xas/solaris10/gtk/libX11-1.3.2/lib/libX11.so.6.3.0:XPending+0x64
fc78ecd8 /pf/m/m222086/xas/solaris10/gtk/gtk-2.20.1/lib/libgdk-x11-2.0.so.0.2000.1:gdk_x11_pixmap_get_drawable_impl+0x1d80

Hmpf, recompiling the whole stuff with debugging symbols enabled, I
did a deeper investigation: 

1.) If I disable the Xrender-Extension on my local Linux xorg-server,
everything just works fine (firefox started remotely through ssh -X,
just as before).

2.) With a breakpoint at XGrabButton and sniffing the X traffic with
xmon, I concluded that X_GrabButton really never has been
sent. Indeed, the error return has its opcode set to 28 which is
X_GrabButton (I had a breakpoint at _XError to verify this fact). 
The first very strange thing, at least for me.

3.) Now, I thought: Well, certainly the reason is not that my xerver's
maximum request length has been hit, but that it's "an internal Xlib
error". I dumped the X traffic (RENDER extension enabled again) with
xmon, extracted the client requests, converted the hexadecimal stuff
to binay stuff and piped it into a self-written analysis program that
just print's out each major/minor opcode and the requests' lengths.
If there's sth wrong with the length's (encoded within the 3th and 4th
request of each request), I thought, I'll certainly have too few or
too many bytes left at the end. I had not.  But another strange thing
happended: Although XRenderQueryVersion gets called two times and
sends the corresponding requests (through XRenderQueryFormats) and
even receives replies (major version 0, minor version 10), as I've
seen in gdb, the request with major opcode 154 and minor opcode 0 does
not show up in the traffic (although many other 154-requests do).  154
is definitely the RENDER major opcode here as xdpyinfo and gdb tell me
(Xrender.c:425, in XRenderQueryFormats). The next very strange thing
for me. Well, who knows what xmon does exactly when I tell it to
capture raw traffic...  
Btw: The maximum length of those requests examined is 261304 bytes,
xdpyinfo tells me
the maximum request size: 16777212 bytes


Do you've got any idea? I'm completely stuck. I really don't know how
to get around that issue...

Thank you very much!
Wishes

Nicolai