[Bug 741754] New: Proposal for a "framecache" element.

[GStreamer (bugzilla.gnome.org)]

https://bugzilla.gnome.org/show_bug.cgi?id=741754
  GStreamer | gst-plugins-bad | git

           Summary: Proposal for a "framecache" element.
    Classification: Platform
           Product: GStreamer
           Version: git
        OS/Version: Linux
            Status: UNCONFIRMED
          Severity: enhancement
          Priority: Normal
         Component: gst-plugins-bad
        AssignedTo: gstreamer-bugs at lists.freedesktop.org
        ReportedBy: mathieu.duponchelle at epitech.eu
         QAContact: gstreamer-bugs at lists.freedesktop.org
     GNOME version: ---


Filing against -bad because I am realistic :)

Warning:

This proposal assumes correcly behaving elements. It isn't concerned with
closed / open GOP as it deals with already decoded frames.

Problem:
--------

One wants to be able to query "recently" decoded frames from a pipeline without
needing to decode them again.

One wants to be able to scrub forward *and* backward in an "as smooth as
possible" way.

Use cases:
----------

For now, in such a pipeline:

decodebin ! some_filter random_property=X ! sink

if one wants to update random_property in the PAUSED state and
get the same frame updated in the sink, one has to seek flush+accurate.

That's hardly ideal.

In the paused state again, if one wants to step to the next frame,
same deal, flush+accurate, decodebin crunches the same thing over again.

Proposed solution:
------------------

On modern-day workstations, a lot of RAM is readily available.
A full HD decoded picture in the RGBA colorspace is 8294400 bytes, ie roughly
7.91 megabytes.
On my machine with 16 GB of RAM, I'd happily sacrifice 1GB, keeping in mind
that unused RAM is bad RAM.

This represents around 130 frames that could be stored.

The proposal here is to implement a framecache element to act as storage,
active only in the PAUSED state, in passthrough mode in the PLAYING state.

This element would be inserted immediately after the decodebin.

One could imagine using multiple frame caches after each filter to maximize
smoothness, but this would introduce complexity to decide who would handle the
seek, as far as I can tell changing properties of elements in the decodebin can
not affect the actual output image (or can it ?), whereas changing a property
on a filter can.

Seek handling "negotiation" is thus out of scope and excluded of this proposal.

Part of the complexity in the proposed design will be due to the requirement of 
having both forward *and* backward scrub seeking working smoothly.

How to ensure that the first seek to a given region remains as fast as without
the framecache, and at the same time store frames as soon as possible in the
surrounding region.

Solution: the frame cache intercepts the seek, and returns TRUE. Caller is
responsible for checking the bus for ASYNC_DONE, or if the seek actually
failed, a custom "async seek failed" that need not be defined in this proposal.
(does this already exist ?)

All the following operations take place in a task running on the sinkpad of the
framecache.

The framecache element will then request a
SEEK_KEY_UNIT | GST_SEEK_FLAG_SNAP_BEFORE upstream, which should be as fast
as a normal seek (check -> is it true ?). start and stop are set at the
required start.

Upon reception of the segment, the element knows where the previous keyframe
was, takes note of that. It also knows the next position before which to seek
when filling itself in the backward direction.

It then waits for EOS and pushes the last buffer it received downstream. All
the buffers are stored, a counter is incremented to mark the number of buffers
stored before the current position.

The element then requests a SEEK_KEY_UNIT | GST_SEEK_FLAG_SNAP_AFTER (note that
some time is "wasted" in that first operation, ideally one would be able to
specify "KEY_UNIT | SNAP_BEFORE" for the start and "KEY_UNIT | SNAP_AFTER" for
the stop in the first seek but meh.

Buffers are stored blabla.

stop is noted, one now knows the next position to seek when filling the buffer
in the forward direction. A counter is incremented to mark the number of
buffers stored after the current position.

Things go on until the sum of "before" and "after" buffers exceeds a "soft
limit".

When one receives a new seek, two solutions are possible:

1) The buffer is already stored in the framecache's memory -> it is sent,
counters for "before" and "after" buffers are updated,
if needed some buffers are discarded on one side and seeks performed on the
other side to maintain "symetry".

2) The buffer is not stored in the framecache's memory: immediately forward a
new flushing seek upstream, repeating the process that happened on the first
seek, the same update as in the other case is performed.

Flushes of course have to be handled appropriately by the element, this isn't
detailed here.

As it is an element with only one sink and one source pad possibilities for
race conditions seem low at first sight ^^

-- 
Configure bugmail: https://bugzilla.gnome.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
You are the assignee for the bug.