[Mesa-dev] [PATCH 1/2] mesa/vbo: Fix scaling issue in 10-bit signed normalized packing.

[Marek Olšák]

On Wed, Oct 31, 2012 at 7:45 PM, Kenneth Graunke <kenneth at whitecape.org> wrote:
> On 10/15/2012 09:45 AM, Ian Romanick wrote:
>>
>> On 10/14/2012 12:02 PM, Kenneth Graunke wrote:
>>>
>>> For the 10-bit components, the divisor was incorrect.  A 10-bit signed
>>> integer can represent -2^9 through 2^9 - 1, which leads to the following
>>> ranges:
>>>
>>>         (float)value.x          -> [ -512,  511]
>>> 2.0F * (float)value.x          -> [-1024, 1022]
>>> 2.0F * (float)value.x + 1.0F   -> [-1023, 1023]
>>>
>>> So dividing by 511 would incorrectly scale it to approximately:
>>> [-2.001956947, 2.001956947].  To correctly scale to [-1.0, 1.0], we need
>>> to divide by 1023.
>>
>>
>> This is a very annoying part of the desktop GL specification: there are
>> two different ways to convert 10-bit and 2-bit normalized integers to
>> float.  GLES3 "fixes" this, I believe, by having only one conversion.
>> Can you double-check these changes against the GLES3 rules?
>
>
> Sigh.  You're right...this code implements equation 2.2
>
>    f = (2c + 1)/(2^b - 1).                                (2.2)
>
> which is mandatory and correct for desktop OpenGL 4.1 and earlier. Starting
> with GL 4.2 and GLES 3, the spec changed to instead require equation 2.3:
>
>    f = max{c/(2^(b-1) - 1), -1.0}                         (2.3)
>
> It also explicitly marks this as a change in behavior.  So we'll need to
> conditionalize it based on the current context API and version, I guess.
> Ugh.

Interesting. FYI, the piglit tests (draw-vertices-2101010 and attribs)
currently expect that equation 2.2 is used.

Marek