[cairo] Planar YUV support
Arjen Nienhuis
a.g.nienhuis at gmail.com
Tue Mar 9 09:00:52 PST 2010
Hi,
Your works looks really useful! Did you make read/write support for
all these formats? That means it can speedup things like JPEG
enc/decoding right?
Minor nitpick:
I think pixman_color_space_t is not the right name. It's ambiguous
because color space can also mean sRGB vs AdobeRGB. What about
'encoding'?
On Tue, Mar 9, 2010 at 5:09 PM, Benjamin Otte <otte at redhat.com> wrote:
> Hey,
>
> Here's an update on my work on YUV planar support, also known as "what
> we got from the gstreamer-cairo hackfest". I've reworked my experimental
> code from back then to incorporate the things we agreed on. It is not
> complete yet, but only misses details. So if you're a maintainer of the
> libraries in question, now is a good time for a review and complaints or
> issues with the general design of the code.
>
> The code can be found in these 3 places:
> http://cgit.freedesktop.org/~company/cairo/log?h=planar
> http://cgit.freedesktop.org/~company/pixman/log?h=planar
> http://cgit.freedesktop.org/~company/gst-plugins-cairo
> I constantly rebase and update it while I work on it, so that the final
> merge contains a proper set of patches.
>
> I'd like to merge this stuff to the respective master branches soon
> after Cairo 1.10 is out, so that it gets enough testing and exposure
> before it's contained in the next major Cairo release, which I hope will
> happen before the end of September.
>
> I'll now give a short description of what these patches do, what the
> issues are and what's missing.
>
> pixman:
>
> The code adds a pixman_color_space_t enum that can be used to specify
> the color space the values are in. Valid values so far are ARGB,
> ARGB_UNMULTIPLIED, YCBCR_SD, YCBCR_HD and YCBCR_JPEG. Various new
> pixman_format_code_t's were added to match the planar formats used in
> YUV. Finally a new constructor for images was added:
> pixman_image_t *
> pixman_image_create_planar (pixman_format_code_t format,
> pixman_color_space_t color_space,
> int width,
> int height,
> unsigned int num_planes,
> uint32_t **bits,
> int *rowstrides);
> This constructor combines the above features to add support for all the
> image representations commonly used. So if you want to create a
> pixman_image for a GdkPixbuf with alpha channel, you'd do:
> uint32_t *bits = (uint32_t *) gdk_pixbuf_get_data (pixbuf);
> int stride = gdk_pixbuf_get_rowstride (pixbuf);
> pixman_image_create_planar (PIXMAN_r8g8b8a8,
> PIXMAN_COLOR_SPACE_ARGB_UNMULTIPLIED,
> gdk_pixbuf_get_width (pixbuf),
> gdk_pixbuf_get_height (pixbuf),
> 1,
> &bits,
> &stride);
> Or to create an image from a GStreamer I420 buffer, you would do:
> for (i = 0; i < 3; i++) {
> bits[i] = (uint32_t *) (buffer_data +
> gst_video_get_component_offset (GST_VIDEO_FORMAT_I420,
> i, width, height));
> strides[i] = gst_video_format_get_row_stride (GST_VIDEO_FORMAT_I420,
> i, width);
> }
> pixman_image_create_planar (PIXMAN_y420,
> PIXMAN_COLOR_SPACE_YCBCR_SD,
> width, height,
> 3, bits, strides);
> That is roughly anything you need to know as a user of pixman.
>
> Details missing in the implementation that I intend to fix:
> - The conversion matrices for HD and JPEG color spaces are still
> missing. The ones I randomly copied so far were all wrong. And I was too
> lazy to do the math myself yet. (I blame COG for getting them wrong.)
> - Writing for i420 is not implemented. I haven't found a way to
> implement it yet that passed my not-ugly test.
> - There is no fast paths at all for the YUV-related formats yet. I
> certainly want to add the ones that are needed, but that requires some
> real life tests first.
> - Getting the fetchers right for subsampled formats and different
> filters.
> Things that I don't intend to fix before merging, but would be happy to
> see others have a go at:
> - formats I consider "unimportant" such as 4:1:1 video or the other
> 4:2:0 chroma varieties other than the MPEG1 one that GStreamer uses and
> that I've implemented.
>
>
> cairo:
>
> The only change visible to users of Cairo will be the addition of the
> cairo_color_space_t enum, additions to the cairo_format_t enum that
> expose the required pixman formats and a new constructor:
> cairo_public cairo_surface_t *
> cairo_image_surface_create_planar (cairo_format_t format,
> cairo_color_space_t color_space,
> int width,
> int height,
> unsigned int n_planes,
> char ** data,
> int * strides);
> This constructor matches the pixman constructor above. Even the examples
> would look very similar. For the GdkPixbuf you'd do:
> uint32_t *bits = (uint32_t *) gdk_pixbuf_get_data (pixbuf);
> int stride = gdk_pixbuf_get_rowstride (pixbuf);
> cairo_image_surface_create_planar (CAIRO_FORMAT_RGBA32,
> CAIRO_COLOR_SPACE_ARGB_UNMULTIPLIED,
> gdk_pixbuf_get_width (pixbuf),
> gdk_pixbuf_get_height (pixbuf),
> 1,
> &bits,
> &stride);
> In fact, I'm using a gdk patch right now that does exactly this. (I also
> have a WebKit patch that does this to improve <canvas> and svg filters.)
> To create an image from a GStreamer I420 buffer, you would do:
> for (i = 0; i < 3; i++) {
> bits[i] = (uint32_t *) (buffer_data +
> gst_video_get_component_offset (GST_VIDEO_FORMAT_I420,
> i, width, height));
> strides[i] = gst_video_format_get_row_stride (GST_VIDEO_FORMAT_I420,
> i, width);
> }
> cairo_image_surface_create_planar (CAIRO_FORMAT_PLANAR_420,
> CAIRO_COLOR_SPACE_YCBCR_SD,
> width, height,
> 3, bits, strides);
> And that is exactly what gst-plugins-cairo does.
>
> I actually had to write quite a bit of code in Cairo to make this work,
> because a lot of the backends assume they get image surfaces in one of
> the 3 common format: A8, RGB24 or ARGB32. And this of course is no
> longer true.
> To be sure to catch all the cases, I removed the image_surface's data
> and stride members. With planar images, they don't make any sense
> anyway. That of course causes a lot of compilation failures and I
> haven't yet fixed all of them. In particular the backends that don't
> work on my computer (Most notably win32 and CoreGraphics) or
> experimentaql haven't been fixed.
>
> Another thing I haven't decided on and need some input is what
> backend(s) to focus on for accelerated uploads (and downlaods) of YUV
> image data. The obvious choices are GL and XCB, but GL is still
> experimental and XCB suffers from X not having any support for YUV (no,
> xv doesn't count). (Before anyone asks: In the long run both should be
> well supported, but I still need something to start with).
>
> gst-plugins-cairo:
>
> There are no real changes to the API since the hackfest. I just updated
> the internals to conform to the new API and thread-safety guarantees
> from Cairo. (And that was mostly deleting code).
> My first goal here is to make the gstreamer sink the default videosink.
> And after that work on improved embedding of video streams (think
> browsers, but also Clutter) and a live editing framework for video
> editors like Pitivi. (annotating Byzanz screencasts should be trivial!)
> What I've been wondering about though is what subset of currently
> supported YUV/RGB formats is a sane subset. It doesn't make a lot of
> sense to me to support all those crazy formats when no one uses them.
> I'm in particular thinking about Y41B, NV21, and all those weird RGB
> formats. Can we agree on a list of formats that we want to support
> "natively" as opposed to "convert asap to a native format"?
>
>
> So, that's the current state of the gstreamer-cairo saga, now it's
> Søren's, Carl's, Chris' and others turn to comment on it if they wanna
> avoid me merging it as is. ;)
>
> Cheers,
> Benjamin
>
> --
> cairo mailing list
> cairo at cairographics.org
> http://lists.cairographics.org/mailman/listinfo/cairo
More information about the cairo
mailing list