[gst-devel] Gst-python object cleanup

[Edward Hervey]

On Fri, 2009-10-30 at 00:39 -0700, Ron McOuat wrote:
> Thanks Edward, the links helped a lot. I do have a signal watch and was 
> removing it in the message callback once the EOS arrived on the filesink 
> element bus which was inserted in order to detect EOS. I was able to 
> determine the Python object count is not increasing using some gc calls 
> from the article referred to by the second link you gave me. Moving the 
> application to Karmic from Jaunty and re-arranging some of the callback 
> code has improved the memory use rate. The old record bin cleanup was in 
> the EOS message callback and the new record bin was allocated right 
> after injecting the EOS into the old record bin so there was a race 
> between the two. The old and new bins had different class variable 
> references bound to keep them apart so I am reasonably sure I wasn't 
> stomping on the wrong things with a shared reference. I now create the 
> new record bin in the EOS callback after the old one is cleaned up 
> making the operations serial. There is a queue in front to accept frames 
> while this is taking place so the front end doesn't lose data or block.
> 
> Watching the python process with gnome-system-monitor it seems to jump 
> up about a MByte once in a while and not in a monotonic once per cycle 
> way. It is like the Python arena gets fragmented somehow and so the 
> virtual machine grabs a new chunk to work with - highly speculative but 
> pymalloc front ends system malloc allocating large chucks. To put it in 
> perspective, the stress testing cycles the application through about 2 
> months of work every 24 hours of test run time, with the latest changes 
> memory consumption will probably be acceptable.

  If you've exhausted your investigations with the gc module (and the
related tools/scripts), you could go down to the C layer and use tools
like massif (from valgrind) to figure out where the memory consumption
happens over time. Unfortunately they removed the graphical visualiser
in some recent versions.

> 
> BTW I would like to say thanks for gstreamer, it is a great multimedia 
> processing library and is improving steadily.

  yay \o/

>  The Python bindings make 
> it even easier to work with - I am relatively new to the language, it is 
> so clean and concise - I come from Java since 2000, C, C++ since 1983 
> and Fortran/Assembler since 1972.

  yay again :) You've come a long way.

    Edward

> 
> Ron
> 
> Edward Hervey wrote:
> >   The description of how you get rid of your pipelines seems correct.
> >
> >   There's only one other thing that *could* go wrong, is if you're using
> > a signal watch on the bus, you should remove it (i.e.
> > gst.Bus.remove_signal_watch()).
> >
> >   If you've made sure that's correct, I'd then suggest to start tracing
> > the leaks at the python level. There's a couple of good articles/tools
> > to do that:
> >   http://www.lshift.net/blog/2008/11/14/tracing-python-memory-leaks
> >   http://mg.pov.lt/blog/hunting-python-memleaks
> >   And you can also have a look at how debugging memory leaks is done in
> > the gst-python testsuite (gst-python/tests/common.py)
> >
> >     Edward
> >
> >   
> >> Thanks for any suggestion.
> >>
> >> Ron McOuat
> >>
> >>
> >>
> >> ------------------------------------------------------------------------------
> >> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> >> is the only developer event you need to attend this year. Jumpstart your
> >> developing skills, take BlackBerry mobile applications to market and stay 
> >> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> >> http://p.sf.net/sfu/devconference
> >> _______________________________________________
> >> gstreamer-devel mailing list
> >> gstreamer-devel at lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
> >>     
> >
> >
> > ------------------------------------------------------------------------------
> > Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> > is the only developer event you need to attend this year. Jumpstart your
> > developing skills, take BlackBerry mobile applications to market and stay 
> > ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> > http://p.sf.net/sfu/devconference
> > _______________________________________________
> > gstreamer-devel mailing list
> > gstreamer-devel at lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
> >
> >   
> 
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay 
> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> http://p.sf.net/sfu/devconference
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel