[Bug 701287] New: GNonLin API should be revisted before release of first 1.X versions

[GStreamer (bugzilla.gnome.org)]

https://bugzilla.gnome.org/show_bug.cgi?id=701287
  GStreamer | gnonlin | git

           Summary: GNonLin API should be revisted before release of first
                    1.X versions
    Classification: Platform
           Product: GStreamer
           Version: git
        OS/Version: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: Normal
         Component: gnonlin
        AssignedTo: gstreamer-bugs at lists.freedesktop.org
        ReportedBy: nicolas.dufresne at collabora.co.uk
         QAContact: gstreamer-bugs at lists.freedesktop.org
     GNOME version: ---


While porting PiTiVi and GES to GStreamer 1.0 we have identified few API
changes that we would like to be done in GNonLin before the next major release.
I'm using this bug to open the discusion and API changes proposal. If anyone
would like other changes that we have not thought about, it would be the time
to do so.

Rate adjustment:

Current API suggest that rate can be ajusted by setting a media-duration
different from the duration of the source. The idea was that rate adjustment
would be done through the rate parameter of the segment. The problem with that
approach is that you need an element at rendering to change that rate back to
normal and add/remove frame and samples, a job normally done by the sink
element. Also, we think it's not possible with that to implement smooth rate
ramp up as available on many video editors.

Proposed solution is to drop media-duration and rate propertie, and favour
element that do the rate ajustment by removing or adding data as needed. On the
audio side, such element already exist and is called pitch (scaletempo could
also support this mode of operation). On video side, we found that videorate
can be modified easily to also do that. In the long term, an OpenCV elements
could be added to use the optical flow algorithm to reconstruct the missing
images during slow motion. We would most likely need something to make rampup
implementation a bit easier, as changing the rate usually change the output
duration and position does not change in constant anymore. This could be an
element in GNL, with few preset rampup model (and eventually ability to set a
rampup path), that would return proper position and duration, or a new base
class (though it would probably clash with other baseclass like videofilter and
such).


Commit base changes:

Currently, the only safe way to modify a composition, or an object, is to pause
the pipeline first. We think that this limitation eliminates many use cases for
GNonLin. Few video editors let you create a draft of your video by applying
multi-cam technic. Also people could want to eventually consider doing live
video production, which obviously require being able to change the composition
in playing state (and increase compositon duration as we go).

The proposes solution is to make GNL fully commit base. Objects, Operations and
Compositon would keep a copy of modified values, and only apply those changes
when a commit is executed. This will allow making the changes (or group of
changes) fully atomic without having to pause the pipeline. Another benifit is
that thread safety will be greatly enhanced in general, resulting in less risk
of randomness. Obviously, we would commit the state when moving from READY to
PAUSED state in order to keep the simple cases simple.

API Whise, this would introduce a commit action into GnlObject. Doing commit on
a GnlComposition, will automatically commit sources. Sources with let say
composition inside of it will not be commited, to provide greater control over
sub-pipelines. The update property in GnlComposition would be dropped.


Rename media-start as inpoint :

This is probably the simplest change, after discussion and comparision between
GES and GNL, we concluded that inpoint is a more correct term and will be
better understood. It was also confusing to have twice the word start in our
discussions.


Support for stream-id:

One thing that we can't do easily at the moment it to select 1 of many audio
streams. GStreamer 1.0 has introduces stream-id into stream-start event to
enable that. We would add a stream-id property to GnlSource to support that. We
would keep the caps property, but move it to GnlSource, as it make no sense in
GnlComposition and GnlOperation.


MAXUINT32 expandables:

This was probably how expandables worked prior to the expandable property.
Using MAXUINT32 has some limitation and which are fully covered by the
expandable property. We would drop this support, which will enabel some code
cleanup.


Better sinks property in GnlOperation:

We want to find a better name for the sinks property of GnlOperation. We would
also like to enable ranges to fully support elements that handle from n to m
elements (like the frie0r elements, that can have 2 or 3 inputs, videomixer and
adder, that supports from 1 to infinit input, etc). These detailed cannot be
guest by the operation itself. The naming is to be defined.


Automatic GAP filling:

Currently doing automatic gap filling using expandable is limiting and complex.
This will eventually be solved by adding a callback that would let the
application know that a GAP has been found, and what type of gap, when is this
gap and what's the duration. This way the application can do fancier stuff
depending if this is about filling a fixed input video mixer or if it's a
regular cap in the timeline. This is no blocker for this bug, as adding API can
be done later, though, not filling a GAP might become an error. For this
reason, we may introduce errors when gaps are detected at runtime, we should at
least document it in this API change.

I think this is it for now, please feel free to add more, our main focus is
what would change the API and ABI.

-- 
Configure bugmail: https://bugzilla.gnome.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
You are the assignee for the bug.