[cairo] Planar YUV support

Arjen Nienhuis a.g.nienhuis at gmail.com
Tue Mar 9 09:00:52 PST 2010


Your works looks really useful! Did you make read/write support for
all these formats? That means it can speedup things like JPEG
enc/decoding right?

Minor nitpick:

I think pixman_color_space_t is not the right name. It's ambiguous
because color space can also mean sRGB vs AdobeRGB. What about

On Tue, Mar 9, 2010 at 5:09 PM, Benjamin Otte <otte at redhat.com> wrote:
> Hey,
> Here's an update on my work on YUV planar support, also known as "what
> we got from the gstreamer-cairo hackfest". I've reworked my experimental
> code from back then to incorporate the things we agreed on. It is not
> complete yet, but only misses details. So if you're a maintainer of the
> libraries in question, now is a good time for a review and complaints or
> issues with the general design of the code.
> The code can be found in these 3 places:
>  http://cgit.freedesktop.org/~company/cairo/log?h=planar
>  http://cgit.freedesktop.org/~company/pixman/log?h=planar
>  http://cgit.freedesktop.org/~company/gst-plugins-cairo
> I constantly rebase and update it while I work on it, so that the final
> merge contains a proper set of patches.
> I'd like to merge this stuff to the respective master branches soon
> after Cairo 1.10 is out, so that it gets enough testing and exposure
> before it's contained in the next major Cairo release, which I hope will
> happen before the end of September.
> I'll now give a short description of what these patches do, what the
> issues are and what's missing.
> pixman:
> The code adds a pixman_color_space_t enum that can be used to specify
> the color space the values are in. Valid values so far are ARGB,
> pixman_format_code_t's were added to match the planar formats used in
> YUV. Finally a new constructor for images was added:
> pixman_image_t *
> pixman_image_create_planar (pixman_format_code_t      format,
>                            pixman_color_space_t      color_space,
>                            int                       width,
>                            int                       height,
>                            unsigned int              num_planes,
>                            uint32_t                **bits,
>                            int                      *rowstrides);
> This constructor combines the above features to add support for all the
> image representations commonly used. So if you want to create a
> pixman_image for a GdkPixbuf with alpha channel, you'd do:
> uint32_t *bits = (uint32_t *) gdk_pixbuf_get_data (pixbuf);
> int stride = gdk_pixbuf_get_rowstride (pixbuf);
> pixman_image_create_planar (PIXMAN_r8g8b8a8,
>                            PIXMAN_COLOR_SPACE_ARGB_UNMULTIPLIED,
>                            gdk_pixbuf_get_width (pixbuf),
>                            gdk_pixbuf_get_height (pixbuf),
>                            1,
>                            &bits,
>                            &stride);
> Or to create an image from a GStreamer I420 buffer, you would do:
> for (i = 0; i < 3; i++) {
>  bits[i] = (uint32_t *) (buffer_data +
>      gst_video_get_component_offset (GST_VIDEO_FORMAT_I420,
>                                      i, width, height));
>  strides[i] = gst_video_format_get_row_stride (GST_VIDEO_FORMAT_I420,
>                                                i, width);
> }
> pixman_image_create_planar (PIXMAN_y420,
>                            PIXMAN_COLOR_SPACE_YCBCR_SD,
>                            width, height,
>                            3, bits, strides);
> That is roughly anything you need to know as a user of pixman.
> Details missing in the implementation that I intend to fix:
> - The conversion matrices for HD and JPEG color spaces are still
> missing. The ones I randomly copied so far were all wrong. And I was too
> lazy to do the math myself yet. (I blame COG for getting them wrong.)
> - Writing for i420 is not implemented. I haven't found a way to
> implement it yet that passed my not-ugly test.
> - There is no fast paths at all for the YUV-related formats yet. I
> certainly want to add the ones that are needed, but that requires some
> real life tests first.
> - Getting the fetchers right for subsampled formats and different
> filters.
> Things that I don't intend to fix before merging, but would be happy to
> see others have a go at:
> - formats I consider "unimportant" such as 4:1:1 video or the other
> 4:2:0 chroma varieties other than the MPEG1 one that GStreamer uses and
> that I've implemented.
> cairo:
> The only change visible to users of Cairo will be the addition of the
> cairo_color_space_t enum, additions to the cairo_format_t enum that
> expose the required pixman formats and a new constructor:
> cairo_public cairo_surface_t *
> cairo_image_surface_create_planar (cairo_format_t       format,
>                                   cairo_color_space_t  color_space,
>                                   int                  width,
>                                   int                  height,
>                                   unsigned int         n_planes,
>                                   char **              data,
>                                   int *                strides);
> This constructor matches the pixman constructor above. Even the examples
> would look very similar. For the GdkPixbuf you'd do:
> uint32_t *bits = (uint32_t *) gdk_pixbuf_get_data (pixbuf);
> int stride = gdk_pixbuf_get_rowstride (pixbuf);
> cairo_image_surface_create_planar (CAIRO_FORMAT_RGBA32,
>                                   CAIRO_COLOR_SPACE_ARGB_UNMULTIPLIED,
>                                   gdk_pixbuf_get_width (pixbuf),
>                                   gdk_pixbuf_get_height (pixbuf),
>                                   1,
>                                   &bits,
>                                   &stride);
> In fact, I'm using a gdk patch right now that does exactly this. (I also
> have a WebKit patch that does this to improve <canvas> and svg filters.)
> To create an image from a GStreamer I420 buffer, you would do:
> for (i = 0; i < 3; i++) {
>  bits[i] = (uint32_t *) (buffer_data +
>      gst_video_get_component_offset (GST_VIDEO_FORMAT_I420,
>                                      i, width, height));
>  strides[i] = gst_video_format_get_row_stride (GST_VIDEO_FORMAT_I420,
>                                                i, width);
> }
> cairo_image_surface_create_planar (CAIRO_FORMAT_PLANAR_420,
>                                   CAIRO_COLOR_SPACE_YCBCR_SD,
>                                   width, height,
>                                   3, bits, strides);
> And that is exactly what gst-plugins-cairo does.
> I actually had to write quite a bit of code in Cairo to make this work,
> because a lot of the backends assume they get image surfaces in one of
> the 3 common format: A8, RGB24 or ARGB32. And this of course is no
> longer true.
> To be sure to catch all the cases, I removed the image_surface's data
> and stride members. With planar images, they don't make any sense
> anyway. That of course causes a lot of compilation failures and I
> haven't yet fixed all of them. In particular the backends that don't
> work on my computer (Most notably win32 and CoreGraphics) or
> experimentaql haven't been fixed.
> Another thing I haven't decided on and need some input is what
> backend(s) to focus on for accelerated uploads (and downlaods) of YUV
> image data. The obvious choices are GL and XCB, but GL is still
> experimental and XCB suffers from X not having any support for YUV (no,
> xv doesn't count). (Before anyone asks: In the long run both should be
> well supported, but I still need something to start with).
> gst-plugins-cairo:
> There are no real changes to the API since the hackfest. I just updated
> the internals to conform to the new API and thread-safety guarantees
> from Cairo. (And that was mostly deleting code).
> My first goal here is to make the gstreamer sink the default videosink.
> And after that work on improved embedding of video streams (think
> browsers, but also Clutter) and a live editing framework for video
> editors like Pitivi. (annotating Byzanz screencasts should be trivial!)
> What I've been wondering about though is what subset of currently
> supported YUV/RGB formats is a sane subset. It doesn't make a lot of
> sense to me to support all those crazy formats when no one uses them.
> I'm in particular thinking about Y41B, NV21, and all those weird RGB
> formats. Can we agree on a list of formats that we want to support
> "natively" as opposed to "convert asap to a native format"?
> So, that's the current state of the gstreamer-cairo saga, now it's
> Søren's, Carl's, Chris' and others turn to comment on it if they wanna
> avoid me merging it as is. ;)
> Cheers,
> Benjamin
> --
> cairo mailing list
> cairo at cairographics.org
> http://lists.cairographics.org/mailman/listinfo/cairo

More information about the cairo mailing list