Huge memory leak sometime after starting a pipeline
Sergei Vorobyov
sergei.vorobyov at facilitylabs.com
Sun Nov 9 05:23:03 PST 2014
Nothing seems working...
The program I posted (with autovideosink/vaapisink) after about 5 minutes
running on nvidia with vdpau increases memory usage from 0.3% to 2.6% ans
starts to complain "not enough resources" and the images blacks out:
*** Cycle #12 through all media files in /home/fl/ads
playing /home/fl/ads/488.jpg
dynamically linking decodebin on pad src_187 to imagefreeze ...
dynamic link succeeded!
received message type=1, name=eos
GST_MESSAGE_EOS
playing /home/fl/ads/242.jpg
dynamically linking decodebin on pad src_188 to imagefreeze ...
dynamic link succeeded!
received message type=1, name=eos
GST_MESSAGE_EOS
playing /home/fl/ads/402.png
dynamically linking decodebin on pad src_189 to imagefreeze ...
dynamic link succeeded!
play time fully expired!
playing /home/fl/ads/494.jpg
dynamically linking decodebin on pad src_190 to imagefreeze ...
dynamic link succeeded!
vdpau_video: VdpVideoSurfaceCreate(): status 23: The system does not have
enough resources to complete the requested operation at this time.
vdpau_video: VdpVideoSurfaceCreate(): status 23: The system does not have
enough resources to complete the requested operation at this time.
vdpau_video: VdpVideoSurfaceCreate(): status 23: The system does not have
enough resources to complete the requested operation at this time.
received message type=1, name=eos
GST_MESSAGE_EOS
playing /home/fl/ads/240.jpg
dynamically linking decodebin on pad src_191 to imagefreeze ...
dynamic link succeeded!
vdpau_video: VdpVideoSurfaceCreate(): status 23: The system does not have
enough resources to complete the requested operation at this time.
vdpau_video: VdpVideoSurfaceCreate(): status 23: The system does not have
enough resources to complete the requested operation at this time.
vdpau_video: VdpVideoSurfaceCreate(): status 23: The system does not have
enough resources to complete the requested operation at this time.
received message type=1, name=eos
GST_MESSAGE_EOS
playing /home/fl/ads/243.jpg
dynamically linking decodebin on pad src_192 to imagefreeze ...
dynamic link succeeded!
vdpau_video: VdpVideoSurfaceCreate(): status 23: The system does not have
enough resources to complete the requested operation at this time.
vdpau_video: VdpVideoSurfaceCreate(): status 23: The system does not have
enough resources to complete the requested operation at this time.
vdpau_video: VdpVideoSurfaceCreate(): status 23: The system does not have
enough resources to complete the requested operation at this time.
received message type=1, name=eos
GST_MESSAGE_EOS
...
This is with GStreamer v1.2.4.0.
Tomorrow I will check GStreamer 1.4.3 (with probably newer vdpau libraries;
maybe vaapisink is better)
On Sat, Nov 8, 2014 at 11:03 PM, Sergei Vorobyov <
sergei.vorobyov at facilitylabs.com> wrote:
> Confirmed: on NVidia with nvidia driver and vdpau vaapisink also leaks
> memory (while ximagesink does not)
>
> Bottom line: deduction beats valgrind, always
>
> On Sat, Nov 8, 2014 at 8:30 PM, Sergei Vorobyov <
> sergei.vorobyov at facilitylabs.com> wrote:
>
>> RE: memory leak in vaapisink
>>
>> I think, this leak is somewhere in vaapisink.
>>
>> Indeed, when I installed video acceleration packages for AMD Radeon (with
>> the latest fglrx proprietary driver):
>>
>> apt-get install vainfo xvba-va-driver libvba-glx1 libva-egl1
>>
>> the same program I posted yesterday started to leak memory with vaapisink
>> and autovideosink (which I guess falls back to vaapisink) on AMD Radeon the
>> same way as previously on Intel's HD IGP with video acceleration.
>>
>> It's highly unlikely that two independent companies produce similarly
>> leaking drivers. Plausible conclusion: vaapisink is to be blamed.
>>
>> (note that my program with ximagesink DOES NOT leak memory)
>>
>>
>>
>> On Fri, Nov 7, 2014 at 5:26 PM, Sergei Vorobyov <
>> sergei.vorobyov at facilitylabs.com> wrote:
>>
>>> > Regarding my comment to simplify your pipeline... try to reproduce the
>>> > leak without rendering video to the screen and just a fakesink. And try
>>> > to reproduce the leak with a much simpler source but displaying on the
>>> > screen. Dividing a problem into subproblems does not only help with
>>> > quicksort ;)
>>>
>>> 1. > try to reproduce the leak without rendering video to the screen
>>> and just a fakesink...
>>>
>>> Do you understand that if you render nothing (fakesink) then evidently
>>> the driver does not work and hence does not leak? I checked it nevertheless
>>> for your pleasure (if I misunderstand something) to confirm the obvious: no
>>> leaks with fakesink.
>>>
>>> 2. > try to reproduce with a much simpler source
>>>
>>> Can't be any simpler: cyclically playing static .jpg and/or .png files
>>> leaks on Intel with vaapisink or autovideosink (but not on NVidia and
>>> Radeon), even repeatedly playing one and the same small randomly chosen
>>> file.
>>>
>>> 3. > simplify your pipeline...
>>>
>>> Can't be any simpler: filesrc location=...! decodebin ! imagefreeze !
>>> videodecode ! vaapisink (implemented in C; see below), just
>>> dynamically changing the location of a .jpg or .png file.
>>>
>>> 4. > Dividing...
>>>
>>> I even know and use more sophisticated things than divide-and-conquer.
>>> But I think with Intel all of it is pointless. They would never answer or
>>> react. The best thing I figured is to dump Intel's HW altogether. Strange
>>> that none of you (mostly working on Intel's) noticed massive memory leak
>>> problems. Do you ever make endurance testing for a few hours or days?
>>>
>>> Here's a simple program in question. Try it like this
>>>
>>> test vaapisink
>>>
>>> (or test <whateversink> for comparison)
>>>
>>> on Intel's (I tried Bay Trail and HD4400 IGPs) and watch with top or
>>> htop, for 10-30 minutes or a few hours.
>>>
>>> test without arguments defaults to autovideosink
>>>
>>> test ximagesink does not leak on Intel.
>>>
>>> test <whateversink> does not leak on NVidia nor AMD Radeon
>>>
>>> /*
>>> * Adjust the MEDIA_DIR constant as explained below
>>> * Compile with:
>>> * gcc -std=c11 -Wall -pthread test.c -o test `pkg-config --cflags
>>> --libs glib-2.0 gstreamer-1.0 gstreamer-pbutils-1.0 gstreamer-video-1.0`
>>> * Run, giving a sink factory name as an argument, like "test vaapisink"
>>> and enjoy
>>> */
>>> #include <stdio.h>
>>> #include <stdlib.h>
>>> #include <glib.h>
>>> #include <gst/gst.h>
>>> /*
>>> * This program implements cyclically playing all image files in
>>> MEDIA_DIR
>>> * through the pipeline
>>> * filesrc location=... ! decodebin ! imagefreeze ! videoconvert | <sink>
>>> * by setting the location property to different files in MEDIA_DIR
>>> */
>>> // Modify as needed. Directory should contain only image files: .jpg,
>>> .png, ..., one or more.
>>> #define MEDIA_DIR "/home/fl/ads"
>>>
>>> static GstBus *bus;
>>> static GstMessage *message;
>>> static GstElement *pipeline, *filesrc, *decodebin, *videoconvert,
>>> *imagefreeze, *sink;
>>> static gchar const *videosink;
>>>
>>> static void dynamic_pad_link_cb (GstElement *decbin, GstPad *srcpad,
>>> gpointer imgfreeze) {
>>> gchar *pad_name = gst_pad_get_name (srcpad);
>>> g_print ("dynamically linking %s on pad %s to %s ...\n",
>>> GST_OBJECT_NAME (decbin), pad_name, GST_OBJECT_NAME
>>> (imgfreeze));
>>> g_free (pad_name);
>>> GstCaps *caps = gst_pad_query_caps (srcpad, NULL);
>>> GstPad *sinkpad = gst_element_get_compatible_pad ((GstElement*)
>>> imgfreeze, srcpad, caps);
>>> GstPadLinkReturn pad_link_outcome = gst_pad_link (srcpad, sinkpad);
>>> if (pad_link_outcome == GST_PAD_LINK_OK)
>>> g_print ("dynamic link succeeded!\n");
>>> else {
>>> g_print ("dynamic link failed with code %d\n", pad_link_outcome);
>>> exit (-1);
>>> }
>>> gst_object_unref (sinkpad);
>>> gst_caps_unref (caps);
>>> }
>>>
>>> static void build_pipeline () {
>>> pipeline = gst_pipeline_new ("main_pipeline");
>>> filesrc = gst_element_factory_make ("filesrc", "filesrc");
>>> decodebin = gst_element_factory_make ("decodebin", "decodebin");
>>> videoconvert = gst_element_factory_make ("videoconvert",
>>> "videoconvert");
>>> imagefreeze = gst_element_factory_make ("imagefreeze",
>>> "imagefreeze");
>>> sink = gst_element_factory_make (videosink, videosink);
>>> if (!sink) {
>>> g_print ("failed to create %s\n", videosink);
>>> exit (-1);
>>> }
>>> gst_bin_add_many (GST_BIN (pipeline), filesrc, decodebin,
>>> imagefreeze, videoconvert, sink, NULL);
>>> gst_element_link (filesrc, decodebin);
>>> g_signal_connect (decodebin, // instance to connect signal to
>>> "pad-added", // detailed signal
>>> G_CALLBACK (dynamic_pad_link_cb), // assoc callback
>>> imagefreeze); // passed as the 3rd arg to the callback
>>> gst_element_link (imagefreeze, videoconvert);
>>> gst_element_link (videoconvert, sink);
>>> bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
>>> }
>>>
>>> static void play_media_file (gchar const *fname) {
>>> gchar *full_file_name = g_strdup_printf (MEDIA_DIR G_DIR_SEPARATOR_S
>>> "%s", fname);
>>> g_object_set (filesrc, "location", full_file_name, NULL);
>>> g_print ("playing %s\n", full_file_name);
>>> if (gst_element_set_state (pipeline, GST_STATE_PLAYING) ==
>>> GST_STATE_CHANGE_FAILURE) {
>>> g_print ("failed to set pipeline to PLAYING\n");
>>> exit (-1);
>>> }
>>> message = gst_bus_timed_pop_filtered (bus, 5 * GST_SECOND,
>>> (GstMessageType)(GST_MESSAGE_EOS |
>>>
>>> GST_MESSAGE_SEGMENT_DONE |
>>>
>>> GST_MESSAGE_ERROR));
>>> if (message == NULL)
>>> g_print ("play time fully expired!\n");
>>> else {
>>> GstMessageType msg_type = GST_MESSAGE_TYPE (message);
>>> gchar const *name = gst_message_type_get_name (msg_type); // a
>>> reference to the static name of the message, don't free
>>> g_print ("received message type=%d, name=%s\n", msg_type, name);
>>> if (msg_type & GST_MESSAGE_EOS)
>>> g_print ("GST_MESSAGE_EOS\n");
>>> else if (msg_type & GST_MESSAGE_ERROR)
>>> g_print ("GST_MESSAGE_ERROR, bad %s\n", full_file_name);
>>> else
>>> g_print ("UNEXPECTED message type %s\n", name);
>>> gst_message_unref (message); // only when != NULL !!!
>>> message = NULL;
>>> }
>>> if (gst_element_set_state (pipeline, GST_STATE_READY) ==
>>> GST_STATE_CHANGE_FAILURE) {
>>> g_print ("failed to set pipeline to state READY\n");
>>> exit (-1);
>>> }
>>> g_free (full_file_name);
>>> gst_element_unlink (decodebin, imagefreeze); // dynamically linked
>>> each time, independently of this unlink
>>> }
>>>
>>> gint main (gint argc, gchar** argv) {
>>> gst_init (&argc, &argv);
>>> guint major, minor, micro, nano;
>>> gst_version (&major, &minor, µ, &nano);
>>> g_print ("GStreamer v%d.%d.%d.%d\n", major, minor, micro, nano);
>>> videosink = (argc > 1)? argv[1] : "autovideosink";
>>> g_print ("using %s\n", videosink);
>>> build_pipeline ();
>>> for (gint i=1; ; i++) {
>>> g_print ("*** Cycle #%d through all media files in %s\n", i,
>>> MEDIA_DIR);
>>> GDir *dir = g_dir_open (MEDIA_DIR, 0, NULL);
>>> gchar const *file_name;
>>> while ((file_name = g_dir_read_name(dir))) // file_name must not
>>> be modified or freed
>>> play_media_file (file_name);
>>> g_dir_close(dir);
>>> }
>>> }
>>>
>>>
>>>
>>> On Thu, Nov 6, 2014 at 11:47 AM, Sebastian Dröge <
>>> sebastian at centricular.com> wrote:
>>>
>>>> On Do, 2014-11-06 at 11:33 +0100, Sergei Vorobyov wrote:
>>>> > Sorry to say but the universal (on this list) recipe to use valgrind
>>>> is
>>>> > useless, like a hammer to drive screws.
>>>> >
>>>> > Point is, to see "leaks", you need to stop your application, in which
>>>> case
>>>> > GStreamer produces thousands (miles) of messages of the kind:
>>>> > [...]
>>>> >
>>>> > even though you make a clean exit with _unrefs of all kinds and
>>>> gst_deinit
>>>> > (), and the application does not leak at all (if you don't stop it
>>>> and run
>>>> > indefinitely).
>>>> >
>>>> > Sure (by the end of the day) you can suppress any kind of messages
>>>> you want
>>>> > to ignore, but this hardly approaches you to the solution of the
>>>> problem.
>>>> > Should you take the above message seriously or ignore? Can you ask
>>>> > GStreamer developers "take seriously or ignore?" about each of
>>>> 100.000 such
>>>> > messages?
>>>>
>>>> The big leaks will be at the top, the small ones at the bottom. If
>>>> there's nothing big at the top you have a problem and valgrind's
>>>> memcheck tool won't be useful for you. And as I said last time already,
>>>> valgrind is not solving all problems but it's a good start... and if it
>>>> doesn't help (it does help in like 90% of the cases!) it still gives you
>>>> hints at where *not* to waste your time looking. Debugging memory leaks
>>>> is not trivial, use any help that you can get.
>>>>
>>>> Also try to simplify your pipeline to get a minimal testcase that
>>>> reproduces the problem. And also see my mention of massif, which will be
>>>> useful while the application is running.
>>>>
>>>> > I observed (without valgrind) a curious thing: the same application
>>>> leaks
>>>> > as hell on Intel's HD4400 IGP (NUC) with Intel's driver, but does not
>>>> leak
>>>> > at all on AMD Radeon, nor on NVidia. As I said, in all three cases
>>>> valgrind
>>>> > uselessly reports miles of possibly and definitely lost blocks, which
>>>> makes
>>>> > all three different cases completely indistinguishable.
>>>> >
>>>> > Check if you are using Intel's graphics HW. Maybe this is the reason.
>>>> > Despite the developers claim that GStreamer runs equally good on any
>>>> > hardware.
>>>>
>>>> Sure there might be a driver bug in the Intel driver. Like there can
>>>> also be driver bugs in the others. Apart from those things GStreamer
>>>> does not care at all what hardware you use, and in general most of us
>>>> are using Intel hardware.
>>>>
>>>> Regarding my comment to simplify your pipeline... try to reproduce the
>>>> leak without rendering video to the screen and just a fakesink. And try
>>>> to reproduce the leak with a much simpler source but displaying on the
>>>> screen. Dividing a problem into subproblems does not only help with
>>>> quicksort ;)
>>>>
>>>> --
>>>> Sebastian Dröge, Centricular Ltd · http://www.centricular.com
>>>>
>>>> _______________________________________________
>>>> gstreamer-devel mailing list
>>>> gstreamer-devel at lists.freedesktop.org
>>>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20141109/5fa34daf/attachment-0001.html>
More information about the gstreamer-devel
mailing list