[gst-devel] RFC: libgstrec

[Ronald Bultje]

Hi all,

since avimux seems to actually work now, it's time for me to do what I
wanted to do long ago: write a recording library + application. The
library should give a combination of gstreamer-level and high-level
calls to make recording as simple as possible without losing too much of
the options of the audio/video sources, encoders, muxers and sinks.

Here's the generic idea, I'm open for any comments and haven't started
anything yet, so please make comments if I sound stupid:

Division of plugin (pipelines):
There will be 3 different (sets of) plugins inside libgstrec. The first
one is the video source, the second one is the audio source, the third
one is the pipeline without the video/audio sources, so the possible
video/audio encoder, the video/audio parser or muxer and the sink
(network/stream, disk, gnome-vfs, whatever). The one reason that I need
the audio/video source and the rest of the pipeline to not be directly
connected is A/V sync. I don't want the muxer (like, say, avimux) to do
the sync correction, since that is far beyond what avimux is supposed to
do. avimux can't know these things since it doesn't even know the
framerate at the start. Besides, sometimes, the movie is already in sync
(re-encoding - filesrc ! avidemux ! ... ! avimux ! disksink) and avimux
should by definition do nothing.

Video Source & overlay handling:
The first thing is a video-source. Basically, libgstrec doesn't need to
know anything, an application can set all properties of a video source
itself (concerning capture properties, that is). This will allow new
plugins to be introduced without having to change libgstrec. The one
thing that libgstrec *will* need to cover is overlay handling. This
requires all video-source-plugins to handle overlay similarly. Since all
plugins are currently v4l plugins, this shouldn't be too difficult,
since that's all going to be defined in the v4lelement. I.e., v4lelement
will allow overlay handling. This introduces some caveats, since only
root is allowed to set the video-device's framebuffer. The simplest
solution (I've thought about this a long time, and it's really the
only/best solution) is to create a small application that sets the
framebuffer just before we open the device, just like xawtv's v4l-conf.
It will be set suid root, so that we can set the framebuffer via this
mini-application just before opening the device in the v4lelement.
v4lelement just takes simple arguments like width/height of the video
screen, width/height of windows that cover the recording window etc.
libgstrec will just use the Xwindow ID of the overlay-window and handle
all the repaints and resetting overlay and such. With this, the user
just needs to set the XwindowID, ligstrec will do all the hard work. If
people want to use the framebuffer or console for video-display, they
only need to adjust libgstrec, not the v4lelement plugin (since that
merely takes options for ioctl()s). Here, you see that we lose console
overlay and such, that's just because I want to make certain things,
like video-overlay, as simple as possible.

Please note, overlay handling code still has to be added to
v4lelement... If you have a much better idea for setting the
framebuffer, please let me know ;-).

Audio Source:
video and audio source will be handled pretty much the same. Basically,
you just need to select the audio source and the application can set/get
properties without libgstrec, just as with the video source. This allows
any audio source to be used, without needing a predefined set of audio
sources in libgstrec.

The pipeline:
the rest of the system, like encoders, muxers/parsers and sink. Also
doesn't need predefined values in libgstrec, the application can do all
this itself.

A/V sync:
the pipeline inside libgstrec wil look like this (if we record both
video and audio):

----------------               -----------
| video source |-----.....-----| encoder |\__
----------------      ...      -----------   \---------  --------
                   libgstrec                  | muxer |--| sink |
----------------      ...      ----------- __/---------  --------
| audio source |-----.....-----| encoder |/
----------------               -----------

libgstrec gets audio samples and/or video frames from the sources and
puts them into the rest of the pipeline, after having done A/V sync
correction. A/V sync code will just be copied from lavrec (mjpegtools),
since that's about the simplest yet best-working A/V sync code I've ever
seen :-). It just means that if we lost a frame (i.e. the time
difference between two frames is double what it should be), we send the
frame into the rest of the pipeline twice (compensation for lost
frames). Besides that, we do some audio-samples-per-second and
video-frames-per-second calculations with which we can insert/delete
frames for A/V sync correction. Audio will just serve us for
calculations, we don't duplicate/delete audio samples, only video
frames.

State handling:
as we all know, v4lsrc & co need to be in the READY state to allow
overlay display (since the device is only open()ed at NULL-to-READY).
Capture only starts when we go from READY-to-PAUSE. disksink, for
example, opens a file at NULL-to-READY and closes it at READY-to-NULL.
This gives a problem, because you want to record to new files whenever
you start capturing. Concerning this, it's good that we seperate the
audio/video source and the rest of the pipeline. Basically, libgstrec
will only allow three states, which it handles internally: STOP, PAUSE,
RECORD. STOP means that the video/audio source are in READY and the rest
of the pipeline is in NULL. PAUSE means that everything is in PAUSE
(here, we have a small problem: see below), RECORD means that everything
is in PLAYING state. When avimux and disksink are in PAUSE state, I'm
not sure whether I can push buffers. I hope I can. The thing is, we
might want to record a bit, then pause, record some more, and this will
all go to the same file. However, when we don't go back from
PAUSE-to-PLAYING, we need to rewrite the header to finish the file. So
on PAUSE-to-NULL (in avimux/disksink or whatever combination of
plugins), I need to be able to push one final buffer. If this is not
possible, the pipeline might be set to PLAYING, even when libgstrec's
state is PAUSED. This screws up fps-calculations in avimux so I'll have
to figure something out for that later on (make it a *_set_property() or
something similar). The video/audio source will get one extra function
reset() inside libgstrec, which the application can use if the device or
the framebuffer settings have been changed. For this, we need to close
the device (READY-to-NULL) and directly re-open it (NULL-to-READY) since
these properties can only be taken into effect when we're in the NULL
state.

Video-only vs. Audio-only vs. Video/Audio:
(asked by Dennis aka sienap). it will be possible to record audio-only,
video-only and video/audio combined. The application is responsible for
providing a valid pipeline with audio and/or video input connections on
the beginning of the pipeline and proper parsers/muxers on the other
side of the pipeline. libgstrec will just not do A/V sync when there
only audio or only video (it can, of course, compensate for lost frames
when there's video-only). I'm not sure how useful libgstrec is if we
simply want to record some audio, but maybe it will prove itself useful
:-)...

Conclusion:
With this, libgstrec just serves as a layer between gstreamer and an
end-user application, doing the hardest work (overlay, A/V sync, state
handling) and leaving the finetuning to the application. Of course,
libgstrec will also include a (Gnome2-based) sample application, but
since libgstrec should be glib2 code and a bit of XFree (for the
overlay-window-ID, catching refresh/repaint events), it should just as
well be possible to write a KDE/Qt/Gtk+/whatever-app without
Gnome2-dependency at all.

I'm open for comments and suggestions, but I hope to be able to start
coding on this soon. Of course, it'd be wonderful if I could finish this
before guadec, then we'd have two demo applications instead of one ;-).

Ronald

-- 
-   .-.
-   /V\    | Ronald Bultje <rbultje at ronald.bitfreak.net>
-  // \\   | Running: Linux 2.4.17-XFS and OpenBSD 3.0
- /(   )\  | http://ronald.bitfreak.net/
-  ^^-^^