[Bug 665080] New: API: subtitle overlays for raw and non-raw video buffers

[GStreamer (bugzilla.gnome.org)]

https://bugzilla.gnome.org/show_bug.cgi?id=665080
  GStreamer | gst-plugins-base | git

           Summary: API: subtitle overlays for raw and non-raw video
                    buffers
    Classification: Platform
           Product: GStreamer
           Version: git
        OS/Version: Linux
            Status: NEW
          Severity: enhancement
          Priority: Normal
         Component: gst-plugins-base
        AssignedTo: gstreamer-bugs at lists.freedesktop.org
        ReportedBy: t.i.m at zen.co.uk
         QAContact: gstreamer-bugs at lists.freedesktop.org
     GNOME version: ---


I would like to propose a simple abstraction for representing one or more
subtitle overlay rectangles that should be blended on top of a video.

These can then be blended on top of raw video using the new
gst_video_overlay_composition_blend() convenience API, or they can be attached
to buffers that represent vaapi/vdpau/whatever surfaces for later rendering in
the video sink using the gst_buffer_set_qdata() API.

This follows the design document/draft at [0].


Primary goals:

 - keep it simple and non-intrusive

 - add support for subtitles when using vaapi/vdpau
    decoders

 - provide utility API for overlay elements so they
   don't have to re-implement the same blending
   code everywhere, and then usually only for a
   limited number of video formats.

 - make sure we can easily add more features later


The proposed new API can be found here:
http://cgit.freedesktop.org/~tpm/gst-plugins-base/tree/gst-libs/gst/video/video-overlay-composition.h?h=video-overlay-composition-textoverlay

It adds two new mini-object classes:

 - GstVideoOverlayComposition: contains one or more
   rectangles, may be attached to buffers or blended
   directly onto raw video buffers

 - GstVideoOverlayRectangle: represents an overlay rectangle,
   including ARGB pixel data, overlay position/size/scaling etc.


In the most basic case, an overlay element would simply modify its code to do
something along the lines of:

  comp = get_current_composition (overlay);
  if (overlay->input_is_raw_video) {
      composition_blend (comp, video_buffer);
  } else {
     video_buffer_set_overlay_composition (video_buffer, comp);
  }

The composition is metadata as far as GstBuffer is concerned, and the API
itself follows the same logic, including writability-based-on-refcount.

Even though quite simple, we don't expose the structures directly and require
access via the function API. This is to enforce/ensure writability and proper
locking (multiple threads may be accessing these at the same time), and to
allow for easy extension in future. There's also some "magic" going on under
the hood (like caching of scaled rectangles).

Sequence numbers on both objects will allow for efficient caching/update
mechanisms. Where compositions are attached to buffers they will be attached to
all buffers, not only when they change. This ensures proper state even when
buffers get dropped for some reason (like QoS), and is also more in line with
how we prefer to do it with e.g. caps. The sink/mixer can easily find out
whether anything changed using the sequence numbers.

Future features may include attaching the original text / pango markup in case
the sink (or application) can render that directly; more formats for the
overlay; letting the sinks do the overlaying even for raw video (requires a new
query); support for pre-multiplied alpha; etc.

This will eventually also allow us to have e.g. text rendered at the resolution
of the output surface/display even if the actual video resolution is quite
small and the video gets upscaled for display (currently we'll just blend the
text on top of the video in low resolution and then the text gets upscaled as
well, which results in ugly pixelated subtitles).

Of course there are many alternative ways how one could solve all this, but the
goal was to pick something that's simple, not very intrusive, and doesn't
require major modifications in e.g. playbin2.

[0]
http://cgit.freedesktop.org/gstreamer/gst-plugins-base/tree/docs/design/draft-subtitle-overlays.txt

-- 
Configure bugmail: https://bugzilla.gnome.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
You are the assignee for the bug.