ProcCreatePixmap() crasher

[Adam Jackson]

On Tue, 2008-09-02 at 15:33 -0400, Daniel Drake wrote:
> Hi,
> 
> We're running xserver-1.4.99.906 in the new OLPC OS build. I'm working
> on this bug: http://dev.laptop.org/ticket/8022
> 
> When you open http://wiki.laptop.org/go/Hardware in our firefox-based
> web browser app, X crashes with this trace:
> 
> Program received signal SIGSEGV, Segmentation fault.
> XvDestroyPixmap (pPix=0x0) at xvmain.c:348
> 348	  pScreen = pPix->drawable.pScreen;
> 
> #0  XvDestroyPixmap (pPix=0x0) at xvmain.c:348
> #1  0x0808119f in ProcCreatePixmap (client=0x9450950) at dispatch.c:1342
> #2  0x08085dbf in Dispatch () at dispatch.c:454
> #3  0x0806b63d in main (argc=7, argv=0xbfa7c494, envp=0x0) at main.c:441
> 
> It looks like mozilla requests a large pixmap that cannot be allocated,
> then we try to destroy a NULL pixmap.
> 
> The attached patch solves the X crash by not freeing the NULL pixmap.
> However, with that applied, the application crashes when loading that
> particular page (admittedly not as bad as killing all of X, but hey). It
> dies with these last words:

Attached patch looks correct, I've pushed it to master (and will cherry
it to 1.5, which is coming out any hour now, I promise).

> The program 'rainbow-daemon' received an X Window System error.                 
> This probably reflects a bug in the program.                                    
> The error was 'BadAlloc (insufficient resources for operation)'.                
>   (Details: serial 11493 error_code 11 request_code 53 minor_code 0)            
>   (Note to programmers: normally, X errors are reported asynchronously;         
>    that is, you will receive the error a while after causing it.                
>    To debug your program, run it with the --sync command line                   
>    option to change this behavior. You can then get a meaningful                
>    backtrace from your debugger if you break on the gdk_x_error() function.)    
> 
> Any comments on whether this is a bug in X, mozilla, or something else?

X has an unfortunate coordinate limit at 2^15-1 square.  If the server
is asked to create something bigger it's well within its rights to
refuse.  So the question is what that CreatePixmap request looks like:
if it's asking for something beyond the coordinate limits it's an app
bug, if not it's an X bug.

> Also, Jordan Crouse commented that it is strange that Xv is appearing in
> the call trace here. Does anyone have an explanation for this? My X
> knowledge so is minimal that I don't really know when Xv should or
> should not be used...

It's not strange.  Xv attaches private data to (some) pixmaps, so if the
pixmap being deleted is an Xv pixmap, Xv needs to free that too; but, of
course, the only way to know if it's an Xv pixmap is to ask Xv, so we
just add XvDestroyPixmap to the call chain.

- ajax