[Mesa-users] Mesa XWindows memory leak causing application instability

[Rick Irons]

Hi,

We are encountering sporadic application instability with an XWindows based implementation of Mesa.  The source of the instability appears to be a memory leak.  The details of our investigation are included below.

This issue appears to have been introduced in Mesa 8.0.0 and is present in the current 17.1.2 release.

Our workaround is identified below.  We are interested to know if this is a valid fix or if there is an alternative preferred approach to addressing the problem.

Thanks for any help that you can provide.

Rick



Problem description:
It looks like Mesa is dependent on MIT-SHM extension in the sense that if the MIT-SHM loads, it needs to be left in the same state such that Mesa's call to XShmDetatch() works correctly, with no unintended side-effects.

The order of extension "close" callbacks appear to be driven by a loop in XCloseDisplay() where "Display *" points to a linked-list identified in the "ext_procs" field.

We find that in Mesa's current state, MIT-SHM's close_display extension hook is called _before_ Mesa's extension hook has been called. Thus, MIT-SHM's shm_info has removed/deleted the _XExtDisplayInfo entry associated with the Display * being closed _before_ Mesa has called XShmDetach as part of its close callback.

The effect is to create a new _XExtDisplayInfo pointer that's left pointing to a freed "codes" field. If the _next_ caller of XOpenDisplay() is unlucky enough to a) obtain a recycled address of a "Display *" matching the previous one and b) call XShmQueryVersion(), there is a chance that the "codes" value(s) have been overwritten resulting in XError of BadRequest, BadLength, or hanging (all have been observed).  This appears to be what https://github.com/mesa3d/mesa/commit/0a20051e6da99e91b7bf589ea457c77a8b618f26 alludes to. In fact, I added XShmQueryVersion() back in to xm_api.c, removed the XShmQueryVersion() I'd incorporated into glxgears-based test program, and reproduced this failure:

X Error of failed request:  BadRequest (invalid request code or no such operation)
Major opcode of failed request:  0 ()
Serial number of failed request:  17
Current serial number in output stream:  17

which is exactly what we see when our glxgears-based test program calls XShmQueryVersion() after a few loops.

Reproduction:
See the modified "glxgears" demo. If possible, run on Debian 8.3 and keep hitting escape to close the current gears demo window and advance the loop.

Possible workaround:
If src/mesa/drivers/x11/fakeglx.c references MIT-SHM _before_ registering itself as an extension with the current display, this appears to allow Mesa's call to XShmDetach to occur with no unintended consequence.  MIT-SHM's close callback is called _after_ Mesa has performed its clean up and we see no stranded entries in shm_info associated with the current display.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/mesa-users/attachments/20170609/7ce046e0/attachment-0001.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: glxgears.c
URL: <https://lists.freedesktop.org/archives/mesa-users/attachments/20170609/7ce046e0/attachment-0001.c>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: out.out
Type: application/octet-stream
Size: 3779 bytes
Desc: out.out
URL: <https://lists.freedesktop.org/archives/mesa-users/attachments/20170609/7ce046e0/attachment-0001.obj>