Question about writing a simple playback application

[Nirbheek Chauhan]

On Thu, Oct 14, 2021 at 2:00 AM Matan Safriel via gstreamer-devel
<gstreamer-devel at lists.freedesktop.org> wrote:
> I spent some time digging through the api docs but am not sure whether these items are supported in the library, the FAQ pointed me at this mailing list for asking. I am wondering whether it should be a very robust use case to use gstreamer for a simple video playback application of the following kind ―
>
> I would like to have a video play while it is being programmatically controlled by my own code. The reason for this being that I am developing air-gesture for video control. It is not mandatory for my use case (not in the foreseeable future) that each and every video encoding format on earth would work, I'd gladly do with just a few of the more popular ones or ones that can be converted into a format where the below can be accomplished.
>
> So, here's the list of features I would like to have in my player, more or less in order of importance, the point being that I'd like my code to robustly control these aspects in real-time as the video is playing (my gesture recognition logic would have to trigger them and know their execution status) ...
>
> Control audio loudness and mute / un-mute.
> Pause and un-pause the playback
> Jump back (and forward) arbitrary to arbitrary locations (times) in the video.
> Speed up and slow down the playback compared to the video's original pace (without stopping the playback).
> Stretch ― fast forward and go back from the current video position to arbitrary video points while having a fast play effect between the current and target video locations (times).

You can use playbin:
https://gstreamer.freedesktop.org/documentation/playback/playbin.html?gi-language=c
playbin is just a wrapper around other gstreamer elements + some logic
that is commonly employed for video players. Here's a tutorial:
https://gstreamer.freedesktop.org/documentation/tutorials/playback/playbin-usage.html?gi-language=c
The tutorial doesn't focus on the things you need, but should be
useful as a reference. You can also look at the examples:
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-base/tests/examples/playback/playback-test.c

You can also use GstPlay instead of playbin:
https://gstreamer.freedesktop.org/documentation/play/gstplay.html?gi-language=c
It's a library that is supposed to make writing of video players
simpler. However it is relatively new (replacement for GstPlayer) and
hasn't been documented very well. The best place to look at examples
is the tests: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-bad/tests/check/libs/play.c

To help with your research, I'll just give you some general gstreamer
terminology to search the documentation with:

Audio volume and mute are simple properties on the bin.
Pause / unpause is GST_STATE_PAUSED -> GST_STATE_PLAYING
Jumping back and forth is called seeking (relative or absolute,
key-unit or frame-accurate, etc), see start / stop arguments to
gst_element_seek()
Speed up / slow down is also classified as seeking (playback rate, not
the same as the "video rate" aka framerate), see the "rate" argument
to gst_element_seek()
Stretch is just jump + speed combined, which is why gst_element_seek()
can do both at once

There's a bunch of video players and video editors that use gstreamer
to do all this and more, like Totem
[https://gitlab.gnome.org/GNOME/totem], Pitivi
[https://gitlab.gnome.org/GNOME/pitivi], and
https://github.com/philn/glide (Rust).

I would strongly recommend starting with the tutorials to understand
things: https://gstreamer.freedesktop.org/documentation/tutorials/index.html?gi-language=c

> Really stretch ― zoom into a particular section of the video as it is playing

Due to responsiveness reasons, this is generally handled at the
toolkit level since you can react immediately to user input (f.ex.,
pinch-zoom, ctrl + scrolling, etc). The usual way to do this is to use
a toolkit-specific video sink element, like qmlglsink or gtkglsink,
and zoom at the toolkit widget level in your app. You can also render
to an arbitrary GL texture on any toolkit in your app.

However, if that is not an option in your case, you can use either
compositor (software mixing) or glvideomixer (hardware-accelerated
mixing). You can set height / width / xpos / ypos etc properties on
the sink pad(s) of the element to change the size of the video in the
frame; to zoom out, zoom in, stretch aspect ratio, etc.

> So I wonder whether these items are features of the gstreamer library api. And in case they are, would you be able to point me in the right direction? I am also open to obtaining contractor work for developing such an application (and helping integrate it with a separate gesture recognition module).

There are a number of consultancies that provide services around
GStreamer, in alphabetical order (off the top of my head): Asymptotic
[https://asymptotic.io/], Centricular
[https://www.centricular.com/about/], Collabora
[https://www.collabora.com], Fluendo
[https://fluendo.com/en/multimedia-codecs-gstreamer/], Igalia
[https://www.igalia.com/technology/multimedia]. I work at Centricular.

Cheers,
Nirbheek