gst-plugins-gl proposed plan for 1.0

[Matthew Waters]

Hi,

Here is my take on what should be done regarding the OpenGL plugins in 
GStreamer 1.0

Comments/suggestions welcome.
-------------- next part --------------
OpenGL plugins integration for GStreamer 1.0

Several changes have occured in the move to GStreamer 1.0 and the OpenGL plugins
need to build on the changes.  This document will attempt to outline the changes
in the OpenGL plugins required for use with GStreamer 1.0.

Possible Changes to GStreamer 1.0 API:
--------------------------------------

* Async Marshalling API:

  - GL requires that all state changes (or gl functions) be called in the same 
    thread (we could roll our own within the gl plugins (suboptimal for
    integration with other hardware accelerated API's))
  - could be as simple as a queue + thread that pops and runs functions.
  - Clutter's solution: http://developer.gnome.org/clutter/1.10/clutter-General.html
  * Use GMainLoop/GMainContext/GSource like GDK:
    - X11: http://git.gnome.org/browse/gtk+/tree/gdk/x11/gdkeventsource.c
    - Win32: http://git.gnome.org/browse/gtk+/tree/gdk/win32/gdkevents-win32.c
    - Quartz: http://git.gnome.org/browse/gtk+/tree/gdk/quartz/gdkeventloop-quartz.c
    - Wayland: http://git.gnome.org/browse/gtk+/tree/gdk/wayland/gdkeventsource.c

Changes to the OpenGL structure:
--------------------------------

* VideoContext:

  - Possibility of using VideoContext to define contexts (not a priority)

* GstGLDisplay:

  * Currently, this object provides the following equivelant GStreamer 1.0 objects
    - GstBufferPool accross multiple elements
    - GstMeta (upload, download, gl state)
    - An async API for marshalling functions into a single thread
  - Possibly removed

* GstGLWindow:

  - holds the platform specific functions to deal with GL context creation and deletion
  - sets up the the async function marshaller

* GstGLBuffer:

  - This should be replaced by GstGLMeta

* GstGLMeta (new):

  * GstGLMeta fields:
    - Async thread marshaller object
    - texture id
    - GL state
  * GstVideoMeta fields:
    - width (width of the texture)
    - height (height of the texture)
    - map (gl {up|down}load)
    - unmap (gl {up|down}load)
    - Video format
  * Lifetime:
    - assigned to a buffer by a GstGLBufferPool
    - first element in the gl chain fills it with texture data from
      GstVideoFrame(upload) or gl commands and passes on to the next element
    * next element:
      - filter/effect: does its processing
      - download: takes the texture data from GL and downloads from GL into a GstVideoFrame
      - imagesink: displays the contents of a GL scene
  * Transformations:
    - VAAPI
    - OpenCL
    - VDPAU
    - OVD
    - OpenMAX
    - CUDA
    - Direct3D?
    - ...

* GstGLBufferPool (new):

  - can be shared accross multiple GL elements
  - holds the gl context and gl thread async marshaller object
  - sets the GstGLMeta fields on outgoing buffers
  - has GstGLDisplay/GstGLWindow as a config option

* Caps

  - should be 'video/x-raw' (GLness is described by GstGLMeta)

* Buffers

  * GL to GL:
    - GstGLMeta
  * GL to non-GL / non-GL to GL:
    - GstVideoMeta map/unmap vfunc for upload/download
  * GL to other API / other API to GL:
    - GstMeta transforms

* Base Classes:

  - needed??

Integration with other Gst elements:
------------------------------------

* ClutterVideosink

  - Clutter uses Cogl for its GL rendering which assumes that cogl_* functions
    are the only GL related functions that are called.  Possible solutions are
    the cogl_begin_gl() and cogl_end_gl() pair or unconditionally rendering into
    a framebuffer that Cogl doesn't know/care about and then copying into a CoglTexture.

* QtGLVideosink

  - uses a GLContext directly

* VAAPI

  * GLX:
    - requires RGBA8 OpenGL texture
    - http://cgit.freedesktop.org/libva/tree/va/glx/va_glx.h
  * EGL:
    - http://cgit.freedesktop.org/libva/tree/va/va_egl.h

* OVD:

  - http://developer.amd.com/sdks/amdappsdk/assets/openvideo_decode_api.pdf

* OpenCL:

  - http://sa10.idav.ucdavis.edu/docs/sa10-dg-opencl-gl-interop.pdf

* CUDA:

  - http://www.drdobbs.com/architecture-and-design/222600097