[Bug 664778] New: [souphttpsrc] chunk allocator: memory waste

[GStreamer (bugzilla.gnome.org)]

https://bugzilla.gnome.org/show_bug.cgi?id=664778
  GStreamer | gst-plugins-good | unspecified

           Summary: [souphttpsrc] chunk allocator: memory waste
    Classification: Platform
           Product: GStreamer
           Version: unspecified
        OS/Version: Linux
            Status: UNCONFIRMED
          Severity: major
          Priority: Normal
         Component: gst-plugins-good
        AssignedTo: gstreamer-bugs at lists.freedesktop.org
        ReportedBy: realbrightster at gmail.com
         QAContact: gstreamer-bugs at lists.freedesktop.org
     GNOME version: ---


Hi There,

I observed memory waste related to souphttpsrc’s chunk allocator, plz don’t
misunderstand that this is not a memory leak but just a heavy memory usages.

Let’s assume below configuration in case of playbin2

Basesrc block size = 24KB
Queue2 max-level-bytes = 24MB

The reason for little large “block size” is in order to improve socket
performance.
large socket read size will reduce syscall and get a large data at once, if the
socket buffer has enough packets. So it’s reasonable value, 24KB.

The problem, memory waste, is occurs below circumstance.

1. Network bandwidth is low 
2. So socket buffer has small packets less than block size (24KB). 

Let’s assume that the available socket buffer is always 12KB to make a simple
scenario.

In this scenario,

The buffer which is allocated by “gst_pad_alloc_buffer” always has 24 KB. But
actually, it contains only 12 KB so half of buffer will be wasted.

The worst case is,
Playbin2 links souphttpsrc and queue2 elements if the URI start with HTTP
scheme.
In a queue2, it calculate buffer limit, in other words max-bytes-limit, using
GST_BUFFER_SIZE() not a really allocated size.
- buffer size (GST_BUFFER_SIZE) will modified by got chunk CB to actually
reading size -

Let apply it to our scenario,
buf->length = 12KB, real allocated size = 24KB.
If the queue2 is FULL, actually allocated memory size will be… 48 MB And the 24
MB will be wasted also.

As far as I know, gst_pad_alloc_buffer was applied to reduce memory allocation
(zero-copy?)
But it causes memory waste and it’s a critical factor in embedded system which
has small physical ram.
Actually, I’m very confusing what is the right way. So I’d like to hear what
you think about this case. 
One more question is how many performance improvement you get after apply
zero-copy scheme (chunk allocator)? 

BR,
Davy

-- 
Configure bugmail: https://bugzilla.gnome.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
You are the assignee for the bug.