[gst-devel] Using cairo/pixman for raw video in GStreamer

[Stefan Kost]

Benjamin Otte schrieb:
> Sebastian Dröge <sebastian.droege <at> collabora.co.uk> writes:
>
>   
>> As you can get a cairo_t from every surface, would this mean that all
>> cairo stuff has to support all the pixman surface formats? I mean, would
>> it be possible later to set YUV colors instead of RGB colors in cairo,
>> would it be possible to draw lines or whatever in cairo on some YUV
>> surface, etc?
>> In that case you probably have a lot of work to do ;)
>>
>>     
> Yes, the idea is that you have a function like this:
> cairo_surface_t *gst_cairo_create_surface (GstBuffer *buffer);
> It would look at the buffer's caps and create the right surface from it.
> So writing a colorspace element would look like this:
> static GstFlowReturn
> gst_cairo_colorspace_transform (GstBaseTransform * btrans, GstBuffer * inbuf,
>      GstBuffer * outbuf)
> {
>   cairo_surface_t *in, *out;
>   cairo_t *cr;
>
>   in = gst_cairo_create_surface (inbuf);
>   out = gst_cairo_create_surface (outbuf);
>   cr = cairo_create (out;
>   cairo_set_source (cr, in);
>   cairo_set_operator (cr, CAIRO_OPERATOR_SOURCE);
>   cairo_paint (cr);
>   cairo_destroy (cr);
>   cairo_surface_destroy (in);
>   cairo_surface_destroy (out);
>
>   return GST_FLOW_OK;
> }
> Making that element also a video scaler is one more cairo_scale().
>
> The work required to get this working is pretty small as both pixman and cairo
> are _very_ generic, you basically just need to implement a read_pixel and
> store_pixel vfunc for every format that returns an ARGB guint32 and everything
> will just work.
>   
But having read_pixel/write_pixel will be slow. How can that be
optimized (and/or vectorized). If I recall right "graphics-drivers" for
turbo pascal in dos worked that way and they were slow!

Stefan

> The complexity comes from making it work fast, but even that is not hard, as
> pixman has a very sophisticated acceleration architecture. So it's just writing
> the accelerated versions, which I intend to do for all the ones that already
> exist and then focus on I420 and AYUV.
>
>   
>>> 4) Port elements to use this cairo API
>>> Either add new elements (cairovideotestsrc, cairocolorspace) or add
>>> support for the old ones. While doing this, refine and improve cairo
>>> or pixman, so the elements can be implemented as nicely as possible. A
>>> lot of code inside GStreamer should go away
>>>       
>> That would be the same as gst-plugins-gl works nowadays. I'm all for it,
>> that's definitely a good idea. Not sure if we want a cairo dependency on
>> every video element now already though...
>>
>>     
> We could deprecate the old elements and add new ones or we could improvie the
> old ones. Not sure what the preferred way is. I guess historically gst has gone
> the "write new ones" route.
> Both ways should be equally feasible.
>
>   
>> Which means that cairo/pixman must have a good framework in place to
>> also add new formats easily. If that's given it might make sense, yes.
>>
>> cairo/pixman should also support 8 bit and 16 bit grayscale and the
>> different Bayer formats too then btw.
>>
>>     
> As I said before: Adding new formats is easy, what is hard is making them fast.
>  :)
>
>   
>> Also it would mean, that if you have some codec that decodes into some
>> weird colorformat that is not supported by pixman/cairo yet, that you
>> need to wait for pixman/cairo to support it and gst-plugins-foo to
>> depend on that version or that you have to do conversions internally.
>>
>>     
> Well, there's quite a few formats that are only used by one or two codecs (like
> upside down raw video in AVI). I think it makes a lot of sense to not expose
> them and require separate elements for them.
>
>   
>> All other steps might not make sense not sure what the cairo/pixman
>> people think about supporting random color formats. Also it would mean
>> that GStreamer depends on cairo as a required dependency. But I guess
>> cairo/pixman are at least portable enough to work everywhere.
>>
>>     
> Yeah, the hard dependency of plugins-base on cairo would be necessary. But
> considering cairo is a blessed dep today (textoverlay), it shouldn't be that
> hard to argue?
>
> Benjamin
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
> trial. Simplify your report design, integration and deployment - and focus on 
> what you do best, core application coding. Discover what's new with 
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>