[Pixman] [cairo] Planar YUV support

[Andrea Canciani]

On Wed, Mar 2, 2011 at 3:16 PM, Soeren Sandmann <sandmann at cs.au.dk> wrote:
> Siarhei Siamashka <siarhei.siamashka at gmail.com> writes:
>
>> I'm not a big fan of "let's make this totally universal and future
>> proof" approach if only a very small fraction of this functionality is
>> going to be actually used. Moreover, I suspect that trying to be too
>> general was responsible for slowing down the original "Planar YUV
>> support" plan.
>
> Part of what was derailing that plan may have been my insisting on being
> precise about how these formats fit into the image processing pipeline,
> including how it related to gamma correction and other colorspace
> issues. I still think this is important. However, we can probably leave
> out some specific features, if there is a credible story about how to do
> them later.
>
> The pipeline as it is now:
>
>    1 convert image sample to a8r8g8b8
>    2 extend sample grid in all directions according to repeat
>    3 interpolate between sample according to filter
>    4 transform
>    5 resample
>    6 combine
>    7 store
>
> To add support for potentially subsampled YUV, some additional stages
> have to be inserted before the first:
>
>   -2 interpolate subsampled components of YUV to get the same
>      resolution as the Y plane
>
>   -1 if the format is planar, stitch together components to form YUV
>      pixels
>
>    0 convert to sRGB
>
> Stage -2 is important because the filter used in that interpolation
> should probably be user-specifiable eventually, which has the
> implication that whatever simple support is added first, it needs to be
> clear what filter precisely is being used.
>
> Stage 0 is a color space conversion and need to eventually be
> configurable too, which means it has to be specified which matrix is
> being used.

There are mainly two possibilities for the color space conversion:
a) convert to a "standard" color space in step 0 and convert the
   result of the combining  to the destination color space in step 6.5
b) perform the color conversion source color space -> destination
   space color in step 5.5

SVG seems to use (a), with sRGB and linearRGB as possible color
spaces, but I'd rather have pixman implement the (b) option, which
is consistent with what PDF does.

>
> There is also the question of how to *write* to a subsampled YUV
> image. I don't particularly like read-only image formats, but writing to
> YUV is not simple when you consider clip rectangles, because subsampling
> involves a filter that probably extends outside the clip boxes.
>
> What Andrea is getting at, is presumably how to specify image formats in
> the API. A fully general API like he suggests is perhaps interesting at
> some point, but I agree it shouldn't be prerequisite for getting some
> YUV support in.

Yes, definitely. There are actually two independent problems:
 - extend formats (planar formats, subsampled formats, etc)
 - add color space support

Pixman can already do RGB<->YUV conversion, so adding support for
a new YUV format with only a different memory layout should be quite
simple, but I'd like not to add new public functions just for this special
format (YUV formats are already quite special, being read-only).

Anyway, I think that if the number of formats you want to add is small,
there is no problem with adding them simply as format codes, but if
you find yourself adding a lot of formats which are the same as existing
ones beside for some generalization already included in my proposal,
you might want to consider it again

Example: YV12 can be reused to implement:
 - IYUV and I420 simply by changing the base pointer of each plane
 - NV12 and NV21 by specifying a different pixel_stride
 - IMC1, IMC2, IMC3, IMC4 by specifying appropriate row_strides
   and base pointers for each plane
None of these formats would require changes to pixman if we already
had pixman_format_t

Andrea

>
>> Surely an alternative solution for html5 video fallbacks is to do
>> YUV->RGB conversion of video frames outside of cairo/pixman, and only
>> let cairo/pixman handle the RGB data. The only real problem with this
>> solution is that such multi pass data processing is somewhat slower.
>> We are speaking about at least tens of percents of performance here.
>> And pixman support for native single pass YUV->RGB conversion combined
>> with scaling could help.
>
> YUV conversion is definitely within the scope of pixman, both YUV->RGB
> and RGB->YUV. However, I'm not sure the existing YUV formats are very
> well though out.
>
>
> Soren
>