[Bug 691299] New: Please add an allocator that uses disk storage to provide memory space

[GStreamer (bugzilla.gnome.org)]

https://bugzilla.gnome.org/show_bug.cgi?id=691299
  GStreamer | gstreamer (core) | git

           Summary: Please add an allocator that uses disk storage to
                    provide memory space
    Classification: Platform
           Product: GStreamer
           Version: git
        OS/Version: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: Normal
         Component: gstreamer (core)
        AssignedTo: gstreamer-bugs at lists.freedesktop.org
        ReportedBy: krzysztof.konopko at youview.com
         QAContact: gstreamer-bugs at lists.freedesktop.org
     GNOME version: ---


Created an attachment (id=232911)
 View: https://bugzilla.gnome.org/attachment.cgi?id=232911
 Review: https://bugzilla.gnome.org/review?bug=691299&attachment=232911

Proposed implementation

This is a proposal for a new allocator deriving from GstAllocator:
GstFileMemAllocator.

Rationale
---------
The default allocator uses virtual memory which might be inconvenient when
there's a demand to keep large amount of buffers (e. g. a media ring buffer)
while the amount of physical memory is limited (e. g. an embedded system).

Given that the disk space is available instead, GstFileMemAllocator offers
memory blocks which are mapped to file system blocks in a temporary file. The
idea is to get memory blocks mapped to chunks of a file (mmap() on *nix). This
way it's up to the OS to page them out and in when needed while keeping memory
use to the minimum.

This is particularly useful on systems where physical memory is scarce while
the address space (VM) is still available.

Reusing freed blocks
--------------------
Currently the allocator doesn't reclaim deallocated file block to make it
reusable. This is for simplicity at this stage.

Give the simplicity described above, at least it reclaims the actual disk space
when freeing a memory block (FALLOC_FL_PUNCH_HOLE if supported). This also
means that the allocation should guarantee that the disk space is available if
in the future implementation memory blocks are reusable.

Tests
-----
A number of tests exist. They can be run as follows:

make gst/gstfilememallocator.check
make gst/gstfilememallocator.valgrind

The allocator was also verified in a production environment of an embedded
system (see www.youview.com) as a foundation for a ring buffer for the TS
timeshifer.

Problems
--------

- portability
What is the best approach to not break other platforms support?
It should be possible to add the support for this allocator on other platform
as needed but at the moment ideally the allocator files should be complied only
for *nix system. I'll have a look how to best achieve it unless someone can
suggest me patterns already used (I haven't been looking for these yet).

- name
I don't feel particularly comfortable about the allocator name, i. e.
GstFileMemAllocator. I'll revisit it but if someone can suggest some short and
descriptive name that would be great.

- initialization
I feel that the initialization is a bit awkward, i. e. the allocator needs to
know some information in advance (the file size and where to store it). It sort
of make sense if one think of the allocator as a global resource but I was
wondering whether it'd be useful to have an ability to create them on a whim,
i. e. not necessarily add them to the registry.

Other considerations
--------------------
A potential downside of the allocator used in some scenarios could be
downgrading the performance of the system by paging out pages more critical for
process execution. An example scenario is a ring buffer where there's a
potentially intensive use of memory pages (reads and writes). This should be
dealt with by the application itself (a sit depends on how it uses the
allocator) but the allocator should provide facilities for doing it (additional
API).

Useful web resources:
http://code.google.com/p/pagecache-mangagement/
http://kerneltrap.org/node/7563
http://lkml.indiana.edu/hypermail/linux/kernel/1005.2/01845.html
http://lkml.indiana.edu/hypermail/linux/kernel/1005.2/01953.html


Any comments and suggestion are very welcomed.

-- 
Configure bugmail: https://bugzilla.gnome.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
You are the assignee for the bug.