Accelerated subtitle rendering in 1.0

[Arnaud Vrac]

Hi all,

I'm trying to implement proper subtitle rendering with 1.0 and my
hardware elements. I'm trying to get the following pipeline to work (I
removed the queues for simplicity):

gst-launch filesrc location=file.mkv ! matroskademux name=dmx
    textoverlay name=overlay
    dmx.video_0 ! ! hwvideodec ! overlay.
    dmx.subtitle_0 ! ssaparse ! overlay.
    overlay. ! tee name=t
        t. ! hwvideosink plane=1
        t. ! overlayrender ! hwimagesink plane=2

As you can see I want to avoid blending the subtitles directly on the
video, as it is costly to map video data. Instead I want to render the
video directly on one hardware plane, and the subtitles on another. In
this pipeline example here is what happens:

1/ Composition metadata is added to video buffers by the textoverlay element
2/ The buffers are sent to both:
  - a video sink that renders the video metadata on one plane
  - an image sink that renders the composition metadata on another
plane. The composition metadata is rendered to an ARGB surface by the
generic overlayrender element in this pipeline.

First, what do you think of this pipeline ? Would it enter the
use-cases that are (or will) be handled by subtitleoverlay ? My goal
is to make this work with playbin.

Second, here are the problems are I have with this pipeline:
 - Allocation queries are not forwarded by the tee element, so I had
to hack the textoverlay element to always attach composition metadata
to the video buffers.
 - I have to force async=false on the hwimagesink element, otherwise
the pipeline will not preroll. Maybe I need to do something special
with GAP events or the GAP flag on buffers ?
 - The video API only has blending functions, but in this case my
overlayrender element needs to render the composition metadata
directly without blending, since the allocated frame to render to is
cleared. It would also be nice to have a function to fill a frame with
any color in any pixel format (for clearing for example).

Thanks for your help.

-- 
Arnaud Vrac