[gst-devel] [cairo] Using cairo/pixman for raw video in GStreamer

[Bill Spitzak]

Soeren Sandmann wrote:
> Hi Benjamin,
> 
> All of this sounds good to me. Below are a few comments on how YUV
> formats could be integrated in pixman.
> 
>> 1) Add extensive YUV support to pixman
> 
> Extensive YUV support would be a very useful addition to pixman. Apart
> from the benefits you listed, I think it also makes sense to have YUV
> support in XRender as a more powerful way of doing textured video than
> Xv.
> 
> * Tiles
> 
> Writing one pixel in a chroma subsampled format requires access to a
> 2x2 tile of RGB pixels, but the current general compositing only
> provides one scanline.

This is true of 4:1:1 (and other things with "1" in them but that is the 
only common one). Most of my work with compressed YUV has been with 
4:2:2 which is alternating yuyvyuyv... and can be directly written from 
a single scanline.

> A solution to that may be to move to a tiled architectured where
> general_composite() processes destination tiles instead of
> scanlines. This would require changing all the scanline accessors, but
> hopefully that is a mostly mechanical process.

This probably won't help if the borders of the tiles do not line up with 
the blocks needed. If the input is translated by 1 pixel vertically then 
you will need multiple tiles to write portions of the image.

In general I consider tiled apis to make things unnecessarily 
complicated. The majority of cairo input is packed into an array. You 
either need to require images to be padded out to a multiple of tile 
size, or you need to greatly complicate things with "partial tiles" with 
whatever code is needed to avoid ever addressing the non-existent parts 
of the tiles.

Scanlines are instead trivial to extract from a packed array, and save 
the overhead of having to think about the vertical iterator, allow 
translation and cropping and vertical flipping to be done in-place, and 
require small enough amounts of temporary storage that it tends to be 
done in the cpu cache.

I think storage into 4:1:1 must be done by keeping the previous scanline 
in a temporary buffer and combining them in the scanline processor that 
is writing to the buffer.

> Aside from hopefully solving the subsampling problem, tiles would also
> have better cache behavior for rotated or filtered sources.

No the performance is TERRIBLE for filters. A filter near the edge of a 
tile will require an entire neighboring tile. In scanlines the filter 
always gets only the exact input scanlines needed.

Tiles do help for rotation of giant images, and for drawing a section 
out of the center of an image. But neither of these are common 
operations for Cairo, which really wants to draw images that are smaller 
than the screen fast.

Tiles are also helpful for operations that only change a small portion 
of the image. But I doubt cairo is going to be altered to use a 
referenced-counted set of tiles for all storage, I suspect it is much 
faster to always use memory laid out such that it is in the form that 
the hardware wants. This makes it impossible to use this advantage of tiles.

I think compression of YUV data would have to be done by buffers on I/O. 
Except for detecting literal copies with a translation that allows it, I 
don't think cairo or pixman should attempt to do anything with 
compressed YUV, just like it does not attempt to do anything with 
compressed jpeg data. It should decompose it into YUV channels.

Even then YUV support requires special code as it is not RGB with 
different primaries. Black is .5 in the UV channels. If this is to be 
fast at all all the compositing operations need to be changed. It is not 
as bad as it might look, the interdependence of the cahnnels should 
cancel out of the math, but the calculation of UV will be differnt than 
the ones for Y and RGB.