gstreamer and valgrind

Wed Aug 20 08:06:22 PDT 2014

Hello,

I've composed a sample GStreamer pipeline that leaks, although it doesn't leak as much as in original problem, also it stops leaking before running out of memory.

The command is:

gst-launch-1.0 udpsrc uri="my_ip:5555" ! queue ! decodebin ! videoconvert ! avenc_flv ! flvmux streamable=true ! rtmpsink location="rtmp://rtmp_server:8080/oflaDemo/streamName live=1"

The memory as shown by htop (VIRT/RES/SHR):
1. "No signal" banner: 515M/25212/6308
2. Signal turned on:   517M/26532/6312
3. "No signal" again:  719M/228M/6356
4. Signal turned on:   720M/230M/6356
5. "No signal" again:  913M/424M/6456

On one run I got SIGSEGV on step 3. The backtrace was:
(gdb) back
#0  0x00007f8c248ee03d in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f8c24e2cfe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007f8c24e2d30a in g_main_loop_run ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f8c25378a94 in gst_bus_poll (bus=bus at entry=0x2534b70,
    events=events at entry=GST_MESSAGE_ANY, timeout=<optimized out>)
    at gstbus.c:1082
#4  0x00000000004045a8 in event_loop (pipeline=0x2722040,
    blocking=blocking at entry=1, do_progress=do_progress at entry=0,
    target_state=target_state at entry=GST_STATE_PLAYING) at gst-launch.c:509
#5  0x0000000000403767 in main (argc=17, argv=0x7fff093ce3c8)
    at gst-launch.c:1085

I saved the core in case some other details would be helpful (it's 185M).

When run in Valgrind this pipe grows even without switching the signal. I waited until it got to about 1500M(RES), then did the signal switching. This is what Valgrind says:

==14224== 265,263,534 (63,240 direct, 265,200,294 indirect) bytes in 465 blocks are definitely lost in loss record 3,880 of 3,882
==14224==    at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14224==    by 0x53DA610: g_malloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4000.0)
==14224==    by 0x53F022D: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4000.0)
==14224==    by 0x4E6E36A: gst_buffer_add_meta (gstbuffer.c:1915)
==14224==    by 0x9121760: gst_buffer_add_video_meta_full (gstvideometa.c:231)
==14224==    by 0x91227A8: video_buffer_pool_alloc (gstvideopool.c:202)
==14224==    by 0x4E6F353: do_alloc_buffer.constprop.2 (gstbufferpool.c:272)
==14224==    by 0x4E6F6F5: default_acquire_buffer (gstbufferpool.c:995)
==14224==    by 0x4E70A01: gst_buffer_pool_acquire_buffer (gstbufferpool.c:1099)
==14224==    by 0x7A0D63A: default_prepare_output_buffer (gstbasetransform.c:1577)
==14224==    by 0x7A0F3BA: gst_base_transform_handle_buffer (gstbasetransform.c:2065)
==14224==    by 0x7A0FD00: gst_base_transform_chain (gstbasetransform.c:2201)
==14224==
==14224== 1,392,821,707 (546,744 direct, 1,392,274,963 indirect) bytes in 2,071 blocks are definitely lost in loss record 3,881 of 3,882
==14224==    at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14224==    by 0x53DA610: g_malloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4000.0)
==14224==    by 0x53F022D: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4000.0)
==14224==    by 0x53F076D: g_slice_alloc0 (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4000.0)
==14224==    by 0x912E699: gst_video_encoder_chain (gstvideoencoder.c:1212)
==14224==    by 0x4E97F07: gst_pad_push_data (gstpad.c:3760)
==14224==    by 0x7A0FF08: gst_base_transform_chain (gstbasetransform.c:2237)
==14224==    by 0x4E97F07: gst_pad_push_data (gstpad.c:3760)
==14224==    by 0x4E89F0A: gst_proxy_pad_chain_default (gstghostpad.c:128)
==14224==    by 0x4E97F07: gst_pad_push_data (gstpad.c:3760)
==14224==    by 0x91252BE: gst_video_decoder_clip_and_push_buf (gstvideodecoder.c:2657)
==14224==    by 0x912A447: gst_video_decoder_finish_frame (gstvideodecoder.c:2572)
==14224==
==14224== 1,653,911,262 bytes in 2,658 blocks are indirectly lost in loss record 3,882 of 3,882
==14224==    at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14224==    by 0x53DA610: g_malloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4000.0)
==14224==    by 0x53F022D: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4000.0)
==14224==    by 0x4E627CC: _sysmem_new_block (gstallocator.c:415)
==14224==    by 0x4E6C471: gst_buffer_new_allocate (gstbuffer.c:668)
==14224==    by 0x912275C: video_buffer_pool_alloc (gstvideopool.c:195)
==14224==    by 0x4E6F353: do_alloc_buffer.constprop.2 (gstbufferpool.c:272)
==14224==    by 0x4E6F6F5: default_acquire_buffer (gstbufferpool.c:995)
==14224==    by 0x4E70A01: gst_buffer_pool_acquire_buffer (gstbufferpool.c:1099)
==14224==    by 0x7A0D63A: default_prepare_output_buffer (gstbasetransform.c:1577)
==14224==    by 0x7A0F3BA: gst_base_transform_handle_buffer (gstbasetransform.c:2065)
==14224==    by 0x7A0FD00: gst_base_transform_chain (gstbasetransform.c:2201)
==14224==
==14224== LEAK SUMMARY:
==14224==    definitely lost: 626,733 bytes in 2,548 blocks
==14224==    indirectly lost: 1,657,475,257 bytes in 9,597 blocks
==14224==      possibly lost: 136,289,663 bytes in 600 blocks
==14224==    still reachable: 995,020 bytes in 7,242 blocks
==14224==         suppressed: 0 bytes in 0 blocks
==14224==
==14224== For counts of detected and suppressed errors, rerun with: -v
==14224== Use --track-origins=yes to see where uninitialised values come from
==14224== ERROR SUMMARY: 314 errors from 270 contexts (suppressed: 0 from 0)

-- 
Best regards,
Krzysztof Borowczyk

> -----Original Message-----
> From: gstreamer-devel [mailto:gstreamer-devel-
> bounces at lists.freedesktop.org] On Behalf Of Krzysztof Borowczyk
> Sent: Wednesday, August 20, 2014 3:52 PM
> To: 'Discussion of the development of and with GStreamer'
> Subject: RE: gstreamer and valgrind
> 
> [...]
> 
> > >
> > > GStreamer now works differently
> >
> > Note that the message says that it *might* now take different code
> > paths
> > to ease debugging. As far as I know, it doesn't actually do that in
> > practice, besides printing that info text.
> >
> > >  and I get leaks quite rarely.
> >
> > Are you sure about that? Perhaps it just leaks slower because it runs
> > slower?
> 
> Yes it runs and leaks much slower, but it has been a bit more difficult
> to trigger the leak. The way to trigger it is quite strange (explained
> below).
> 
> The setup is:
> 
> 1. First computer: ffmpeg streaming mpegts to udp://secondComputer:5555
> - ffmpeg grabs input of avermedia grabbing card.
> 2. On the second one the (simplified) pipeline is:
> udpsrc -> queue -> decodebin -> [...] -> avenc_flv -> flvmux ->
> rtmpsink
> 
> I've run similar pipelines with different sources (left of [...])
> without noticing any serious leakage. (shmsink, decklinksrc, filesrc).
> 
> To trigger the leak:
> 1. Start the pipeline with the stream content being video of static
> image (No signal panel from avermedia card on the other computer)
> 2. Start video input on the avermedia, so it's no longer static image.
> 3. After some time turn off the signal, video goes back to static
> image.
> 4. Huge leaking starts now.
> 
> It's a way that triggers the leak 100% of the time. If I stop at point
> 1 or 2 there seems to be no significant leak during 10min test period -
> leaking starts immediately after switching the signal off.
> 
> I suppose this switching of signal on the first computer should have no
> influence on GStreamer, ffmpeg is streaming all the time, just
> different content - do you have any idea why this might happen? I'm
> using GStreamer 1.2.4, were there any significant leak fixes in the
> meantime?
> 
> I will try to prepare a test case with clean GStreamer, without our
> application. In the meantime this is the bottom of Valgrind log without
> G_SLICE=always-malloc, below that one is another one, with G_SLICE set.
> Application was stopped with ctrl-c, unfortunately there is no easy way
> to terminate it normally.
> 
> ==8282== 174,974,310 bytes in 90 blocks are possibly lost in loss
> record 9,438 of 9,440
> ==8282==    at 0x4C2AB80: malloc (in
> /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==8282==    by 0x4E85610: g_malloc (in /lib/x86_64-linux-gnu/libglib-
> 2.0.so.0.4000.0)
> ==8282==    by 0x4E9B22D: g_slice_alloc (in /lib/x86_64-linux-
> gnu/libglib-2.0.so.0.4000.0)
> ==8282==    by 0x63B87CC: _sysmem_new_block (gstallocator.c:415)
> ==8282==    by 0x63C2471: gst_buffer_new_allocate (gstbuffer.c:668)
> ==8282==    by 0x289F275C: video_buffer_pool_alloc (gstvideopool.c:195)
> ==8282==    by 0x63C5353: do_alloc_buffer.constprop.2
> (gstbufferpool.c:272)
> ==8282==    by 0x63C56F5: default_acquire_buffer (gstbufferpool.c:995)
> ==8282==    by 0x63C6A01: gst_buffer_pool_acquire_buffer
> (gstbufferpool.c:1099)
> ==8282==    by 0xB8C663A: default_prepare_output_buffer
> (gstbasetransform.c:1577)
> ==8282==    by 0xB8C83BA: gst_base_transform_handle_buffer
> (gstbasetransform.c:2065)
> ==8282==    by 0xB8C8D00: gst_base_transform_chain
> (gstbasetransform.c:2201)
> ==8282==
> ==8282== 221,634,126 bytes in 114 blocks are indirectly lost in loss
> record 9,439 of 9,440
> ==8282==    at 0x4C2AB80: malloc (in
> /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==8282==    by 0x4E85610: g_malloc (in /lib/x86_64-linux-gnu/libglib-
> 2.0.so.0.4000.0)
> ==8282==    by 0x4E9B22D: g_slice_alloc (in /lib/x86_64-linux-
> gnu/libglib-2.0.so.0.4000.0)
> ==8282==    by 0x63B87CC: _sysmem_new_block (gstallocator.c:415)
> ==8282==    by 0x63C2471: gst_buffer_new_allocate (gstbuffer.c:668)
> ==8282==    by 0x289F275C: video_buffer_pool_alloc (gstvideopool.c:195)
> ==8282==    by 0x63C5353: do_alloc_buffer.constprop.2
> (gstbufferpool.c:272)
> ==8282==    by 0x63C56F5: default_acquire_buffer (gstbufferpool.c:995)
> ==8282==    by 0x63C6A01: gst_buffer_pool_acquire_buffer
> (gstbufferpool.c:1099)
> ==8282==    by 0xB8C663A: default_prepare_output_buffer
> (gstbasetransform.c:1577)
> ==8282==    by 0xB8C83BA: gst_base_transform_handle_buffer
> (gstbasetransform.c:2065)
> ==8282==    by 0xB8C8D00: gst_base_transform_chain
> (gstbasetransform.c:2201)
> ==8282==
> ==8282== 743,315,903 bytes in 377 blocks are still reachable in loss
> record 9,440 of 9,440
> ==8282==    at 0x4C2AB80: malloc (in
> /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==8282==    by 0x4E85610: g_malloc (in /lib/x86_64-linux-gnu/libglib-
> 2.0.so.0.4000.0)
> ==8282==    by 0x4E9B22D: g_slice_alloc (in /lib/x86_64-linux-
> gnu/libglib-2.0.so.0.4000.0)
> ==8282==    by 0x63B87CC: _sysmem_new_block (gstallocator.c:415)
> ==8282==    by 0x63C2471: gst_buffer_new_allocate (gstbuffer.c:668)
> ==8282==    by 0x289F275C: video_buffer_pool_alloc (gstvideopool.c:195)
> ==8282==    by 0x63C5353: do_alloc_buffer.constprop.2
> (gstbufferpool.c:272)
> ==8282==    by 0x63C56F5: default_acquire_buffer (gstbufferpool.c:995)
> ==8282==    by 0x63C6A01: gst_buffer_pool_acquire_buffer
> (gstbufferpool.c:1099)
> ==8282==    by 0xB8C663A: default_prepare_output_buffer
> (gstbasetransform.c:1577)
> ==8282==    by 0xB8C83BA: gst_base_transform_handle_buffer
> (gstbasetransform.c:2065)
> ==8282==    by 0xB8C8D00: gst_base_transform_chain
> (gstbasetransform.c:2201)
> ==8282==
> ==8282== LEAK SUMMARY:
> ==8282==    definitely lost: 77,952 bytes in 274 blocks
> ==8282==    indirectly lost: 223,170,538 bytes in 879 blocks
> ==8282==      possibly lost: 175,129,187 bytes in 2,281 blocks
> ==8282==    still reachable: 772,523,997 bytes in 102,118 blocks
> ==8282==         suppressed: 0 bytes in 0 blocks
> ==8282==
> ==8282== For counts of detected and suppressed errors, rerun with: -v
> ==8282== Use --track-origins=yes to see where uninitialised values come
> from
> ==8282== ERROR SUMMARY: 915 errors from 871 contexts (suppressed: 0
> from 0)
> (END)
> 
> 
> Here a different run with G_SLICE:
> ==13331== 190,527,582 bytes in 98 blocks are possibly lost in loss
> record 9,466 of 9,468
> ==13331==    at 0x4C2AB80: malloc (in
> /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==13331==    by 0x4E85610: g_malloc (in /lib/x86_64-linux-gnu/libglib-
> 2.0.so.0.4000.0)
> ==13331==    by 0x4E9B22D: g_slice_alloc (in /lib/x86_64-linux-
> gnu/libglib-2.0.so.0.4000.0)
> ==13331==    by 0x63B87CC: _sysmem_new_block (gstallocator.c:415)
> ==13331==    by 0x63C2471: gst_buffer_new_allocate (gstbuffer.c:668)
> ==13331==    by 0x287DD75C: video_buffer_pool_alloc
> (gstvideopool.c:195)
> ==13331==    by 0x63C5353: do_alloc_buffer.constprop.2
> (gstbufferpool.c:272)
> ==13331==    by 0x63C56F5: default_acquire_buffer (gstbufferpool.c:995)
> ==13331==    by 0x63C6A01: gst_buffer_pool_acquire_buffer
> (gstbufferpool.c:1099)
> ==13331==    by 0xB8C663A: default_prepare_output_buffer
> (gstbasetransform.c:1577)
> ==13331==    by 0xB8C83BA: gst_base_transform_handle_buffer
> (gstbasetransform.c:2065)
> ==13331==    by 0xB8C8D00: gst_base_transform_chain
> (gstbasetransform.c:2201)
> ==13331==
> ==13331== 217,745,808 bytes in 112 blocks are indirectly lost in loss
> record 9,467 of 9,468
> ==13331==    at 0x4C2AB80: malloc (in
> /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==13331==    by 0x4E85610: g_malloc (in /lib/x86_64-linux-gnu/libglib-
> 2.0.so.0.4000.0)
> ==13331==    by 0x4E9B22D: g_slice_alloc (in /lib/x86_64-linux-
> gnu/libglib-2.0.so.0.4000.0)
> ==13331==    by 0x63B87CC: _sysmem_new_block (gstallocator.c:415)
> ==13331==    by 0x63C2471: gst_buffer_new_allocate (gstbuffer.c:668)
> ==13331==    by 0x287DD75C: video_buffer_pool_alloc
> (gstvideopool.c:195)
> ==13331==    by 0x63C5353: do_alloc_buffer.constprop.2
> (gstbufferpool.c:272)
> ==13331==    by 0x63C56F5: default_acquire_buffer (gstbufferpool.c:995)
> ==13331==    by 0x63C6A01: gst_buffer_pool_acquire_buffer
> (gstbufferpool.c:1099)
> ==13331==    by 0xB8C663A: default_prepare_output_buffer
> (gstbasetransform.c:1577)
> ==13331==    by 0xB8C83BA: gst_base_transform_handle_buffer
> (gstbasetransform.c:2065)
> ==13331==    by 0xB8C8D00: gst_base_transform_chain
> (gstbasetransform.c:2201)
> ==13331==
> ==13331== 799,696,514 bytes in 406 blocks are still reachable in loss
> record 9,468 of 9,468
> ==13331==    at 0x4C2AB80: malloc (in
> /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==13331==    by 0x4E85610: g_malloc (in /lib/x86_64-linux-gnu/libglib-
> 2.0.so.0.4000.0)
> ==13331==    by 0x4E9B22D: g_slice_alloc (in /lib/x86_64-linux-
> gnu/libglib-2.0.so.0.4000.0)
> ==13331==    by 0x63B87CC: _sysmem_new_block (gstallocator.c:415)
> ==13331==    by 0x63C2471: gst_buffer_new_allocate (gstbuffer.c:668)
> ==13331==    by 0x287DD75C: video_buffer_pool_alloc
> (gstvideopool.c:195)
> ==13331==    by 0x63C5353: do_alloc_buffer.constprop.2
> (gstbufferpool.c:272)
> ==13331==    by 0x63C56F5: default_acquire_buffer (gstbufferpool.c:995)
> ==13331==    by 0x63C6A01: gst_buffer_pool_acquire_buffer
> (gstbufferpool.c:1099)
> ==13331==    by 0xB8C663A: default_prepare_output_buffer
> (gstbasetransform.c:1577)
> ==13331==    by 0xB8C83BA: gst_base_transform_handle_buffer
> (gstbasetransform.c:2065)
> ==13331==    by 0xB8C8D00: gst_base_transform_chain
> (gstbasetransform.c:2201)
> ==13331==
> ==13331== LEAK SUMMARY:
> ==13331==    definitely lost: 78,872 bytes in 277 blocks
> ==13331==    indirectly lost: 221,424,134 bytes in 906 blocks
> ==13331==      possibly lost: 190,732,025 bytes in 2,769 blocks
> ==13331==    still reachable: 841,250,822 bytes in 102,333 blocks
> ==13331==         suppressed: 0 bytes in 0 blocks
> ==13331==
> ==13331== For counts of detected and suppressed errors, rerun with: -v
> ==13331== Use --track-origins=yes to see where uninitialised values
> come from
> ==13331== ERROR SUMMARY: 1098 errors from 874 contexts (suppressed: 0
> from 0)
> 
> 
> --
> Best regards,
> Krzysztof Borowczyk
> 
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel