[gst-devel] [cairo] Using cairo/pixman for raw video in GStreamer
spitzak at gmail.com
Mon Sep 14 20:00:19 CEST 2009
Soeren Sandmann wrote:
> Hi Benjamin,
> All of this sounds good to me. Below are a few comments on how YUV
> formats could be integrated in pixman.
>> 1) Add extensive YUV support to pixman
> Extensive YUV support would be a very useful addition to pixman. Apart
> from the benefits you listed, I think it also makes sense to have YUV
> support in XRender as a more powerful way of doing textured video than
> * Tiles
> Writing one pixel in a chroma subsampled format requires access to a
> 2x2 tile of RGB pixels, but the current general compositing only
> provides one scanline.
This is true of 4:1:1 (and other things with "1" in them but that is the
only common one). Most of my work with compressed YUV has been with
4:2:2 which is alternating yuyvyuyv... and can be directly written from
a single scanline.
> A solution to that may be to move to a tiled architectured where
> general_composite() processes destination tiles instead of
> scanlines. This would require changing all the scanline accessors, but
> hopefully that is a mostly mechanical process.
This probably won't help if the borders of the tiles do not line up with
the blocks needed. If the input is translated by 1 pixel vertically then
you will need multiple tiles to write portions of the image.
In general I consider tiled apis to make things unnecessarily
complicated. The majority of cairo input is packed into an array. You
either need to require images to be padded out to a multiple of tile
size, or you need to greatly complicate things with "partial tiles" with
whatever code is needed to avoid ever addressing the non-existent parts
of the tiles.
Scanlines are instead trivial to extract from a packed array, and save
the overhead of having to think about the vertical iterator, allow
translation and cropping and vertical flipping to be done in-place, and
require small enough amounts of temporary storage that it tends to be
done in the cpu cache.
I think storage into 4:1:1 must be done by keeping the previous scanline
in a temporary buffer and combining them in the scanline processor that
is writing to the buffer.
> Aside from hopefully solving the subsampling problem, tiles would also
> have better cache behavior for rotated or filtered sources.
No the performance is TERRIBLE for filters. A filter near the edge of a
tile will require an entire neighboring tile. In scanlines the filter
always gets only the exact input scanlines needed.
Tiles do help for rotation of giant images, and for drawing a section
out of the center of an image. But neither of these are common
operations for Cairo, which really wants to draw images that are smaller
than the screen fast.
Tiles are also helpful for operations that only change a small portion
of the image. But I doubt cairo is going to be altered to use a
referenced-counted set of tiles for all storage, I suspect it is much
faster to always use memory laid out such that it is in the form that
the hardware wants. This makes it impossible to use this advantage of tiles.
I think compression of YUV data would have to be done by buffers on I/O.
Except for detecting literal copies with a translation that allows it, I
don't think cairo or pixman should attempt to do anything with
compressed YUV, just like it does not attempt to do anything with
compressed jpeg data. It should decompose it into YUV channels.
Even then YUV support requires special code as it is not RGB with
different primaries. Black is .5 in the UV channels. If this is to be
fast at all all the compositing operations need to be changed. It is not
as bad as it might look, the interdependence of the cahnnels should
cancel out of the math, but the calculation of UV will be differnt than
the ones for Y and RGB.
More information about the gstreamer-devel