[Mesa-dev] [PATCH 1/2] mesa/vbo: Fix scaling issue in 10-bit signed normalized packing.

[Kenneth Graunke]

On 10/15/2012 09:45 AM, Ian Romanick wrote:
> On 10/14/2012 12:02 PM, Kenneth Graunke wrote:
>> For the 10-bit components, the divisor was incorrect.  A 10-bit signed
>> integer can represent -2^9 through 2^9 - 1, which leads to the following
>> ranges:
>>
>>         (float)value.x          -> [ -512,  511]
>> 2.0F * (float)value.x          -> [-1024, 1022]
>> 2.0F * (float)value.x + 1.0F   -> [-1023, 1023]
>>
>> So dividing by 511 would incorrectly scale it to approximately:
>> [-2.001956947, 2.001956947].  To correctly scale to [-1.0, 1.0], we need
>> to divide by 1023.
>
> This is a very annoying part of the desktop GL specification: there are
> two different ways to convert 10-bit and 2-bit normalized integers to
> float.  GLES3 "fixes" this, I believe, by having only one conversion.
> Can you double-check these changes against the GLES3 rules?

Sigh.  You're right...this code implements equation 2.2

    f = (2c + 1)/(2^b - 1).                                (2.2)

which is mandatory and correct for desktop OpenGL 4.1 and earlier. 
Starting with GL 4.2 and GLES 3, the spec changed to instead require 
equation 2.3:

    f = max{c/(2^(b-1) - 1), -1.0}                         (2.3)

It also explicitly marks this as a change in behavior.  So we'll need to 
conditionalize it based on the current context API and version, I guess. 
  Ugh.