[gst-devel] [RFC] Video blending interface

[Gergely Nagy]

Hi everyone!

This mail is pretty much a draft, or an outline, if one so prefers, of
ideas and requirements I have for a video blending interface. Interface
I say, because there are many ways to blend two streams together, and
all those ways should be separate plugins in my humble opinion (so they
can be chained in weird ways no-one can foresee).

However, before I go on further, I'd like to note that large parts of
what I describe here are already implemented, and probably only need
little tweaking, if at all. If I happened to miss something, I welcome
pointers.

So! Let me start with the requirements:

First of all, for our use-case, we need blending to be as fast as
possible. There are a few optimisations one can implement to achieve
higher speeds, like allowing the two sources to be of different size,
and making sure one can tell the blender plugin that here is this big
background layer, this other one, which is wider, but only 48 pixels
high, please merge it on the background at position -100,30. That would
start blending the `foreground' at its pixel at 100,0 to the background
at 0,30... I don't have to explain that, I think. Given this, it is
quite easy to take a static image (like, a pre-rendered text), and
scroll it through the video stream. Bascially, this would mean that
elements connected to the sink pad of a blender plugin would have a
bounding_box property of some sort (be that a struct *, or x0, y0, x1,
y1 properties, or anything).

Then, there is the requirement that it must properly support the alpha
channel on its srcpad. As far as I see, the current videomixer plugin
always adds some background (black, white or checkered). I don't want
that. I can do that by feeding the blender a background and merging
something onto it.

The third requirement is that blending should be an interface, not one
plugin that does all kinds of blending. There are a few reasons I think
this is a good idea: first, it makes the plugin far less complex;
second, makes it easier to reuse; third, this design still allows a
larger, bin-like plugin which has on-demand sinks, and dispatches the
work to other blenders (together with z-order & stuff).

That's about the requirements, I think.

Lets see how this is different from the currently available videomixer
plugin:

Videomixer supports the xpos/ypos pad properties, which is one half of
the bounding_box approach I need. It also supports an alpha pad
property, which I did not mention, but is important too. However, as far
as I see, it always adds a background to the image at its srcpad. It is
also one single plugin doing too much (imho, at least). It would make
the code simpler and more reusable if different blending methods were
different plugins implementing the same interface. Then videomixer's job
would only be co-ordination.

Now, onto my proposal, the gut of this mail..

I'd like to propose a new interface, labelled `video blending
interface'. Plugins implementing this would take I420 or AYUV as input,
and output AYUV. On their sinkpads, they would have a bounding_box (or
something along those lines) and an alpha property. If any of the
bounding box properties are unset, defaults are used (0,0 -
min(bg_width,fg_width), min (bg_height,fg_height)). They would not have
on-demand sinks, but two static ones: the background, and the
foreground. On the srcpad, the result would always have the same size as
the background (this is because I want to be able to merge a part of a
larger element onto a smaller one). Basically, that's all. Except, I am
not sure how the framerate should be handled...

One idea is to not care about it at all, and on the src pad, always have
a frame rate of 0 (a variant of this is to have the framerate of the
slower sink pad). This would mean that if one of the sinks does not have
new data, the previous buffer we got from it would be reused.

The other is to do as videomixer currently does (which I haven't
inspected, but will do so soonish).

And another is to require both sinks to have the same framerate
(videorate should help there, methinks).

Since most of this is already implemented by videomixer, and all I need
is a few nice tweaks and abstraction of code, I'd like to split up
videomixer into different plugins implementing the same interface, then
reimplementing videomixer as a `frontend' to these.

What do people think about this? Apart from that, that I can't explain
my ideas well enough and talk too much - I already know that, thanks =)

Cheers,
-- 
Gergely Nagy <gergely.nagy at neteyes.hu>
NetEyes Kft.