[Libva] [PATCH] shaders: gen[789]: fix Y coefficient in YUV->RGB conversion

[Lionel Landwerlin]

On 21/10/15 01:57, Zhao Yakui wrote:
> On 10/21/2015 12:48 AM, Lionel Landwerlin wrote:
>> In the following formula for the conversion :
>>
>> R = Clamp ( 1.164(Y-16/255) + 1.596(Cr-128/255))
>> G = Clamp ( 1.164(Y-16/255) - 0.813(Cr-128/255) - 0.392(Cb-128/255))
>> B = Clamp ( 1.164(Y-16/255) + 2.017(Cb-128/255))
>>
>> we must substract 16 (or 16/255 if dealing with [0.0, 1.0] floats) to
>> Y before applying the multiplier coefficent. The shader was missing
>> the substraction.
>
> Hi,
>
>    Thanks for your patch.
>    The issue comes from the confusing formula between YUV and RGB.
> As you see from the below link, it has several different formulas 
> about YUV to RGB.
>     >http://www.equasys.de/colorconversion.html
>
>     And ITU has three standards for the conversion between YUV and RGB.
> (BT601, 709, 2020).
>     Even for BT601, it will also have the different conversion if the 
> footroom/headroom is considered.
>
>     If you hope to fix the conversion issue, I prefer that the shader 
> is based on the generic conversion matrix and the conversion matrix is 
> passed. In such case we can pass the different conversion matrix for 
> the different standard.

Yeah, I was wondering because the driver's code base includes more 
generic shaders that work with a matrix passed from userspace :
http://cgit.freedesktop.org/vaapi/intel-driver/tree/src/shaders/render

Why isn't the driver using those shaders?

Indeed having a matrix would be best. What the current conversion being 
used right now?
Reading your link it seems it's neither BT601 nor BT709.

>
> Thanks
>    Yakui
>> ---
>>   src/shaders/post_processing/gen7/YUV_to_RGB.g4a   | 16 
>> ++++++++++++++++
>>   src/shaders/post_processing/gen7/pl2_to_rgbx.g75b | 16 
>> ++++++++++++++++
>>   src/shaders/post_processing/gen7/pl2_to_rgbx.g7b  | 16 
>> ++++++++++++++++
>>   src/shaders/post_processing/gen8/YUV_to_RGB.g8a   | 16 
>> ++++++++++++++++
>>   src/shaders/post_processing/gen8/pl2_to_rgbx.g8b  | 16 
>> ++++++++++++++++
>>   src/shaders/post_processing/gen9/pl2_to_rgbx.g9b  | 16 
>> ++++++++++++++++
>>   6 files changed, 96 insertions(+)
>