[gst-devel] Using cairo/pixman for raw video in GStreamer

[Sebastian Dröge]

Am Freitag, den 11.09.2009, 19:02 +0200 schrieb Benjamin Otte:

> These are the steps I'd like to propose as a solution:
> 
> 1) Add extensive YUV support to pixman
> The goal is to add an infrastructure so one can support at least the
> formats supported by ffmpegcolorspace today. In fact the ffmpeg
> infrastructure fits pretty well to pixman, but I'm not sure if a
> straight port is acceptable license-wise.
> 
> 2) Add support to Cairo to create surfaces from any pixman image
> I'm not sure how hard this would be, as it basically circumvents
> cairo_format_t - might be possible to hook it into image surfaces or
> might be better to use a different surface backend. But it'd just add
> a single function like cairo_pixman_surface_create (pixman_image_t
> *image);

As you can get a cairo_t from every surface, would this mean that all
cairo stuff has to support all the pixman surface formats? I mean, would
it be possible later to set YUV colors instead of RGB colors in cairo,
would it be possible to draw lines or whatever in cairo on some YUV
surface, etc?
In that case you probably have a lot of work to do ;)

> 3) Add a new caps to gstreamer: video/cairo
> I'm not sure yet about the specific properties required, but certainly
> framerate, width and height are required. Probably an optional
> pixman-format is required, too. Buffers passed in this format only
> contain a reference to a cairo surface in the buffer's data.

(Should really be video/x-cairo). The properties should probably include
the pixman-format too because this way elements can still give
preferences (if they work really better on some format than on another)
or if elements can still only work on a single format (because they need
to fiddle with the bits themself instead of using cairo).

> 4) Port elements to use this cairo API
> Either add new elements (cairovideotestsrc, cairocolorspace) or add
> support for the old ones. While doing this, refine and improve cairo
> or pixman, so the elements can be implemented as nicely as possible. A
> lot of code inside GStreamer should go away

That would be the same as gst-plugins-gl works nowadays. I'm all for it,
that's definitely a good idea. Not sure if we want a cairo dependency on
every video element now already though...

> 6) For next major GStreamer release, remove video/x-raw-*
> The old formats are not needed anymore, they can be removed. All
> elements are ported to the new API.

Which means that cairo/pixman must have a good framework in place to
also add new formats easily. If that's given it might make sense, yes.

cairo/pixman should also support 8 bit and 16 bit grayscale and the
different Bayer formats too then btw.

Also it would mean, that if you have some codec that decodes into some
weird colorformat that is not supported by pixman/cairo yet, that you
need to wait for pixman/cairo to support it and gst-plugins-foo to
depend on that version or that you have to do conversions internally.

> 4) "Cairo/GStreamer developers will not like that"
> In fact, I talked to both of the maintainers and the response in both
> cases was pretty positive, but skeptical about the feasability of such
> a project, mostly fueled by preconceptions about what Cairo or
> GStreamer is and how it works. I consider myself part of both the
> Cairo and GStreamer comunities and know the code in quite some detail
> and I do think it's a very good fit.

Until step 4 of your plan it definitely makes sense and should be done.
Start a GIT repository for gst-plugins-cairo and I'd help you to get the
GStreamer part of things done ;) This could also be done today with just
supporting RGB/RGBA.

All other steps might not make sense not sure what the cairo/pixman
people think about supporting random color formats. Also it would mean
that GStreamer depends on cairo as a required dependency. But I guess
cairo/pixman are at least portable enough to work everywhere.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Dies ist ein digital signierter Nachrichtenteil
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20090912/97249def/attachment.pgp>