[gst-devel] Using cairo/pixman for raw video in GStreamer

[Benjamin Otte]

Sebastian Dröge <sebastian.droege <at> collabora.co.uk> writes:

> As you can get a cairo_t from every surface, would this mean that all
> cairo stuff has to support all the pixman surface formats? I mean, would
> it be possible later to set YUV colors instead of RGB colors in cairo,
> would it be possible to draw lines or whatever in cairo on some YUV
> surface, etc?
> In that case you probably have a lot of work to do ;)
> 
Yes, the idea is that you have a function like this:
cairo_surface_t *gst_cairo_create_surface (GstBuffer *buffer);
It would look at the buffer's caps and create the right surface from it.
So writing a colorspace element would look like this:
static GstFlowReturn
gst_cairo_colorspace_transform (GstBaseTransform * btrans, GstBuffer * inbuf,
     GstBuffer * outbuf)
{
  cairo_surface_t *in, *out;
  cairo_t *cr;

  in = gst_cairo_create_surface (inbuf);
  out = gst_cairo_create_surface (outbuf);
  cr = cairo_create (out;
  cairo_set_source (cr, in);
  cairo_set_operator (cr, CAIRO_OPERATOR_SOURCE);
  cairo_paint (cr);
  cairo_destroy (cr);
  cairo_surface_destroy (in);
  cairo_surface_destroy (out);

  return GST_FLOW_OK;
}
Making that element also a video scaler is one more cairo_scale().

The work required to get this working is pretty small as both pixman and cairo
are _very_ generic, you basically just need to implement a read_pixel and
store_pixel vfunc for every format that returns an ARGB guint32 and everything
will just work.
The complexity comes from making it work fast, but even that is not hard, as
pixman has a very sophisticated acceleration architecture. So it's just writing
the accelerated versions, which I intend to do for all the ones that already
exist and then focus on I420 and AYUV.

> > 4) Port elements to use this cairo API
> > Either add new elements (cairovideotestsrc, cairocolorspace) or add
> > support for the old ones. While doing this, refine and improve cairo
> > or pixman, so the elements can be implemented as nicely as possible. A
> > lot of code inside GStreamer should go away
> 
> That would be the same as gst-plugins-gl works nowadays. I'm all for it,
> that's definitely a good idea. Not sure if we want a cairo dependency on
> every video element now already though...
> 
We could deprecate the old elements and add new ones or we could improvie the
old ones. Not sure what the preferred way is. I guess historically gst has gone
the "write new ones" route.
Both ways should be equally feasible.

> Which means that cairo/pixman must have a good framework in place to
> also add new formats easily. If that's given it might make sense, yes.
> 
> cairo/pixman should also support 8 bit and 16 bit grayscale and the
> different Bayer formats too then btw.
> 
As I said before: Adding new formats is easy, what is hard is making them fast.
 :)

> Also it would mean, that if you have some codec that decodes into some
> weird colorformat that is not supported by pixman/cairo yet, that you
> need to wait for pixman/cairo to support it and gst-plugins-foo to
> depend on that version or that you have to do conversions internally.
> 
Well, there's quite a few formats that are only used by one or two codecs (like
upside down raw video in AVI). I think it makes a lot of sense to not expose
them and require separate elements for them.

> All other steps might not make sense not sure what the cairo/pixman
> people think about supporting random color formats. Also it would mean
> that GStreamer depends on cairo as a required dependency. But I guess
> cairo/pixman are at least portable enough to work everywhere.
> 
Yeah, the hard dependency of plugins-base on cairo would be necessary. But
considering cairo is a blessed dep today (textoverlay), it shouldn't be that
hard to argue?

Benjamin