[Mesa-dev] [PATCH 1/2] mesa/vbo: Fix scaling issue in 10-bit signed normalized packing.

[Brian Paul]

On 10/14/2012 01:02 PM, Kenneth Graunke wrote:
> For the 10-bit components, the divisor was incorrect.  A 10-bit signed
> integer can represent -2^9 through 2^9 - 1, which leads to the following
> ranges:
>
>         (float)value.x          ->  [ -512,  511]
> 2.0F * (float)value.x          ->  [-1024, 1022]
> 2.0F * (float)value.x + 1.0F   ->  [-1023, 1023]
>
> So dividing by 511 would incorrectly scale it to approximately:
> [-2.001956947, 2.001956947].  To correctly scale to [-1.0, 1.0], we need
> to divide by 1023.
>
> Signed-off-by: Kenneth Graunke<kenneth at whitecape.org>
> ---
>   src/mesa/vbo/vbo_attrib_tmp.h | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/src/mesa/vbo/vbo_attrib_tmp.h b/src/mesa/vbo/vbo_attrib_tmp.h
> index d3fc77e..8ad7354 100644
> --- a/src/mesa/vbo/vbo_attrib_tmp.h
> +++ b/src/mesa/vbo/vbo_attrib_tmp.h
> @@ -109,7 +109,7 @@ static inline float conv_i10_to_norm_float(int i10)
>   {
>      struct attr_bits_10 val;
>      val.x = i10;
> -   return (2.0F * (float)val.x + 1.0F) * (1.0F  / 511.0F);
> +   return (2.0F * (float)val.x + 1.0F) * (1.0F  / 1023.0F);
>   }
>
>   static inline float conv_i2_to_norm_float(int i2)


The piglit draw-vertices-2101010 test seems to pass both with and 
without this patch (and I verified that the conv_i10_to_norm_float() 
function is getting called).

I guess the piglit test isn't hitting the issue you're fixing.  Can 
you improve the piglit test?

-Brian