[Mesa-dev] SNORM conversion equation woes

[Jose Fonseca]

D3D10 specifies equation 2.3 (and I'm sure so that D3D9). So probably all hardware can handle it.

I'm surprised that formula 2.2 even exists, and that OpenGL could have been so lax about it.

Jose

----- Original Message -----
> Hello all,
> 
> I've been looking at https://bugs.freedesktop.org/show_bug.cgi?id=59150
> and could use your opinion.
> 
> OpenGL defines two equations for converting from SNORM to floating point
> data.  These are:
> 
> f = (2c + 1)/(2^b - 1)                (equation 2.2)
> f = max{c/2^(b-1) - 1), -1.0}         (equation 2.3)
> 
> Equation 2.2 cannot represent 0.0f, while equation 2.3 can.
> Equation 2.3 has two representations for -1.
> 
> The spec language for which to use is a bit wishy-washy...GL 3.2:
> "In general, [equation 2.2] is used for signed normalized fixed-point
>   parameters in GL commands, such as vertex attribute values."
> "In general, [equation 2.3] is used for signed normalized fixed-point
>   texture or floating point values."
> 
> ARB_vertex_type_2_10_10_10_rev clearly specifies equation 2.2.
> 
> *However*, OpenGL 4.2 and ES 3.0 both remove equation 2.2, instead
> saying to use equation 2.3 in all cases.
> 
> Intel hardware always uses equation 2.3, which provides a dilemma:
> should we just use the native hardware rules, which are fast, or do
> equation 2.2 via shader workarounds?
> 
> The maximum error/difference between the two equations is:
> 
> 16-bit -> 0.00003
> 10-bit -> 0.00195
>   8-bit -> 0.00784
>   2-bit -> 0.66666
> 
> (Data and a script from Keith are available from:)
>   http://people.freedesktop.org/~kwg/snorm/
> 
> Furthermore, since GL 4.2 is supposed to be backwards compatible,
> implementations can silently promote contexts to 4.2, likely giving them
> the new SNORM rules.  Which means applications can't really rely on the
> exact equation used...
> 
> Always using 2.3 seems preferable since it can actually represent 0.
> 
> I am tempted to rip out the code in vbo_attrib_tmp.h that implements
> equation 2.2 and always use 2.3.  That would also require changing
> Piglit's draw-vertices-2101010 test to accept either equation.
> 
> What do you all think?
> 
> Thanks for your time!
>    --Ken
> _______________________________________________
> mesa-dev mailing list
> mesa-dev at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/mesa-dev
>