[Pixman] [cairo] Planar YUV support

[Andrea Canciani]

On Fri, Feb 25, 2011 at 2:00 PM, Siarhei Siamashka
<siarhei.siamashka at gmail.com> wrote:
> On Tuesday 09 March 2010 18:09:57 Benjamin Otte wrote:
>> Hey,
>>
>> Here's an update on my work on YUV planar support, also known as "what
>> we got from the gstreamer-cairo hackfest". I've reworked my experimental
>> code from back then to incorporate the things we agreed on. It is not
>> complete yet, but only misses details. So if you're a maintainer of the
>> libraries in question, now is a good time for a review and complaints or
>> issues with the general design of the code.
>>
>> The code can be found in these 3 places:
>>   http://cgit.freedesktop.org/~company/cairo/log?h=planar
>>   http://cgit.freedesktop.org/~company/pixman/log?h=planar
>>   http://cgit.freedesktop.org/~company/gst-plugins-cairo
>> I constantly rebase and update it while I work on it, so that the final
>> merge contains a proper set of patches.
>
> Hi, what is the current status of this work? Is it still being developed, put
> on hold, or maybe even totally dead by now?
>
> I'm cautiously interested in getting planar YUV format support added to
> pixman/cairo for the purpose of implementing some kind of reasonably
> well performing software fallback solution for html5 video in browsers
> when hardware acceleration is either impossible or too difficult to use.
> And this looks like a much smaller scale task than your work because:
> 1. Just read-only support for planar YUV format is enough
> 2. Image quality is less important than performance, so that the precise
> filtering details or chroma siting can be just ignored
> 3. Probably just only SD color matrix (Rec.  601) would be enough for the start
>
> The only missing thing in pixman API right now is the ability to wrap the
> output of video decoder (3 buffers corresponding to the color planes) into
> pixman_image_t for using it as source image in compositing operations.

I would suggest the creation of a new opaque pixman type, pixman_format_t,
that represents the format of a bits image with more flexibility than
the existing
pixman_format_code_t.

To represent the existing formats (and many more, for example the x1a7r8g8b8
format from https://bugs.freedesktop.org/show_bug.cgi?id=33741) a structure
with the following fields would be sufficient:
 - bpp bits per pixel
 - n_components number of components
 - for each component:
   - shift the offset
   - bits the number of bits of the component

Pixels are to be accesses as uintBPP_t, even if this means accessing
non-existing
uint types (example: uint24_t, used by r8g8b8) and the components shift and bits
fields specify what bits of this value correspond to the component.

A straightforward generalization would be to add endianness as a global field.
Another generalization would be to move bpp to a component-specific pixel_stride
field. A structure with:
 - n_components number of components
 - for each component:
   - shift the offset
   - bits the number of bits of the component
   - pixel_stride the number of bits to be skipped to get the next value
would be sufficient to define both packed and planar formats (with no
downsampling).

Planar bits images would also require the bits_image structure to have
a base pointer and a
row_stride for each plane.

I think that this format/image definition would be very versatile and
extensible (it would
be possible to add downsampling, different number of components,
floating point formats
and more without breaking the API, as long as the format structure is opaque).

Common formats (like packed a8r8g8b8) are easy to recognize and to
handle efficiently
with the existing accessors, while new formats would be possible by
using generic accessors.

I'm afraid that a completely generic accessor might be quite slow, but
that a very limited number
of special cases (like special casing only on the number of
components, or maybe on bpp)
might be sufficient to achieve good performance for a wide range of formats.

>
> Would it be possible to introduce this minor pixman API extension right now,
> possibly in time for pixman 0.22.0 release? And then continue your efforts,
> which are targeted at implementing complete YUV support also suitable for high
> quality video editing purposes (if I understand it correctly).

The very big disadvantage of the change I'm suggesting is that is
nowhere as small
as the addition of few format_codes. Nevertheless, I believe it can be
the correct
way to introduce generic formats in pixman as it provides the opportunity to add
new interesting features and to abstract the concept of pixman format,
so that the
user is actually saying what format it is going to use (r3a5b3g5? it
is not realistic,
but it would be ok. more reasonable examples are the x1a7r8g8b8 mentioned above
and x(8-N)aN with N in {3,4,5,6} which seems to be in use by some
e-Ink devices).

Objections, improvements, comments for/against this idea would be
greatly appreciated.

Andrea