Accelerated subtitle rendering in 1.0

[Tim-Philipp Müller]

On Mon, 2012-08-06 at 20:51 +0200, Arnaud Vrac wrote:

Hi,

> 1/ Composition metadata is added to video buffers by the textoverlay element
> 2/ The buffers are sent to both:
>   - a video sink that renders the video metadata on one plane
>   - an image sink that renders the composition metadata on another
> plane. The composition metadata is rendered to an ARGB surface by the
> generic overlayrender element in this pipeline.
> 
> First, what do you think of this pipeline ? Would it enter the
> use-cases that are (or will) be handled by subtitleoverlay ? My goal
> is to make this work with playbin.

If you want to make it work well with playbin, it would be much easier
if you made a hwsink element that takes care of both the video data and
the subtitle data in one go, i.e. it would render the video data to
plane 1 and upload the subtitle composition data to plane 2 as needed.


> Second, here are the problems are I have with this pipeline:
>  - Allocation queries are not forwarded by the tee element, so I had
> to hack the textoverlay element to always attach composition metadata
> to the video buffers.
>  - I have to force async=false on the hwimagesink element, otherwise
> the pipeline will not preroll. Maybe I need to do something special
> with GAP events or the GAP flag on buffers ?

You need a queue in each branch after tee (at least until we fix tee to
include them..).

>  - The video API only has blending functions, but in this case my
> overlayrender element needs to render the composition metadata
> directly without blending, since the allocated frame to render to is
> cleared. It would also be nice to have a function to fill a frame with
> any color in any pixel format (for clearing for example).

What pixel format do you need for your API ?

The blending funcs are mostly meant as utility functions for overlay
elements that blend the subtitles on top of raw video data directly.

For your use case, the thought was that just getting the ARGB pixels
with

 gst_video_overlay_rectangle_get_pixels_argb() or
 gst_video_overlay_rectangle_get_pixels_unscaled_argb()

would usually be enough. Is your overlay plane in a different format
then?

Cheers
 -Tim