multiple pipelines using splitmuxsink causing Out Of Memory (OOM)

[Tarun Tej K]

I specifically see this issue only when more than 3 pipelines are running
simultaneously.

Posting the OOM killer log:  https://pastebin.com/yzi6BwSx


On Mon, Sep 24, 2018 at 2:49 PM Tarun Tej K <tarun4690 at gmail.com> wrote:

> Hi,
>
>
>
> I want to use the following gstreamer pipeline for multiple IP cameras as
> an infinite process to keep the video buffer of a fixed duration always.
>
> Below is a sample pipeline I am using:
>
>
>
> gst-launch-1.0  -e rtspsrc
> location="rtsp://<user>:<password>@<IP_ADDR>:554/cam/realmonitor?channel=1&subtype=0"
> name = d \
>
>  !  rtph264depay ! h264parse  ! splitmuxsink name = mux
> location=a_video%02d.mp4 max-size-bytes=5000000 max-size-time=2000000000
> max-files=5
>
>
>
> But when I run this pipeline for 4 IP cameras  on the actual target (
> iMX6UL
> <https://www.nxp.com/products/processors-and-microcontrollers/arm-based-processors-and-mcus/i.mx-applications-processors/i.mx-6-processors/i.mx-6ultralite-processor-low-power-secure-arm-cortex-a7-core:i.MX6UL?tab=Design_Tools_Tab>)
> I am facing an excessive memory usage where ultimately the OOM_Killer will
> be started and one of the application gets killed.
>
> I have tried the same scenario on two three other setups below are the
> observations:
>
>
>
>    1. *iMX6UL – 696 MHz** -  512MB RAM* - Custom Built using Yocto (
>    imx-linux-rocko
>    <https://source.codeaurora.org/external/imx/imx-manifest/tree/?h=imx-linux-rocko>)
>    – gstreamer 1.14.2
>
> Ran the 4 pipelines. RAM’s used memory kept growing rapidly, and within 15
> minutes OOM_Killer task has killed few processes.
>
> Also observed that some segment files generated by splitmuxsink are
> growing upto 80MB at times. This should ideally not cross more than 5MB
> (max-size-bytes = 5000000 and max-size-time=2000000000)
>
>
>
>    1. *Raspberry Pi3(Model B) – 1.2 GHz - 1GB RAM* - Custom Built using
>    Yocto (/meta-raspberrypi/tree/recipes-core/images/rpi-basic-image.bb
>    <https://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi/tree/recipes-core/images/rpi-basic-image.bb>)
>    - gstreamer 1.14.2
>
> Ran the 4 pipelines. RAM’s used memory kept growing rapidly to 600 MB
> within 30 minutes, didn’t wait till OOM_Killer to start. Manually stopped
> the processes.
>
>
>
>    1. *x86 – Intel Xeon - 6GB RAM - Ubuntu 18.04* – gstreamer 1.14.1
>    (pre-built)
>
> Ran the 4 pipelines for about 6 hours. Consistent RAM of 96MB used memory
> for these pipelines. No OOM_Kill situation occurred
>
>
>
>    1. *Raspberry Pi3(Model B) – 1.2 GHz - 1GB RAM - debian* – gstreamer
>    1.10.x (pre-built)
>
> Ran the 4 pipelines for about 1 hour. Consistent RAM of 96MB used memory.
> No OOM_Kill situation occurred
>
>
>
>
>
> From these observations it looks like some additional settings related to
> Memory management needs to be done to use the memory efficiently.  Cases 3
> and 4 show that processes actually doesn’t need more than 100MB of memory
> but somehow the memory is being used excessively in cases 1 and 2. Any idea
> what settings will prevent this situation?
>
> I have tried GST_DEBUG with TRACE but unable to deduce anything the flood
> of debug messages. Is there a better way to debug only the buffers flowing
> in and out of splimuxsink?
>
>
> Thanks
>
> Tarun
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20181001/f5dbfdfb/attachment-0001.html>