Out-of-memory error when using multiple threads

[Simon McVittie]

On Wed, 02 Jan 2008 at 17:43:37 -0500, Havoc Pennington wrote:
> To simplify the question for binding authors - is anyone calling 
> anything besides ref, unref, get_data, set_data on DBusConnection inside 
> the add/remove/toggle watch/timeout handlers? This patch would make any 
> other DBusConnection method deadlock.

When I get round to implementing pure-Python main loop integration
(which is the only way to fix several issues in dbus-python), they'll have to
be able to call out to arbitrary Python code, so I'll either need a guarantee
that no locks are held in the callback, or some API I can call to
temporarily release and re-acquire those locks.

Obviously, I'd very much prefer it if this was automatic so I didn't have to
write too much more "hold libdbus at arms' length and poke it very
carefully, otherwise it aborts" code, and if release/reacquire functions
like the ones I describe below are provided, calling them automatically
around the user callbacks would only add a few lines of code to libdbus.

Please don't make a release with the semi-fix unless it's otherwise
critical - I'd rather not have dbus-python have to depend on "libdbus
either less than 1.1.3 or more than 1.2.5" or something, and it's bad
manners to have "the library" (the connection) in an unusable state while
we call a user callback. We don't want users having to remember which
functions they can and can't call from which call back (see: signal
handlers).

A common idiom in CPython goes something like (pseudocode because I
can't remember the actual function names involved):

PyOpaqueThing *state;
state = py_release_locks();
dbus_function_that_can_block();
py_reacquire_locks(state);

In CPython you actually do this with macros:

Py_BEGIN_THREADS
dbus_function_that_can_block();
Py_END_THREADS

which expands to something like:

{ PyOpaqueThing *_py_state = py_release_locks();
dbus_function_that_can_block();
py_reacquire_locks(_py_state); }

Once I'm back in the office tomorrow I'll see if I can find time to
investigate what's needed for a proper fix.

    Simon