[Mesa-dev] SNORM conversion equation woes

[Kenneth Graunke]

Hello all,

I've been looking at https://bugs.freedesktop.org/show_bug.cgi?id=59150 
and could use your opinion.

OpenGL defines two equations for converting from SNORM to floating point 
data.  These are:

f = (2c + 1)/(2^b - 1)                (equation 2.2)
f = max{c/2^(b-1) - 1), -1.0}         (equation 2.3)

Equation 2.2 cannot represent 0.0f, while equation 2.3 can.
Equation 2.3 has two representations for -1.

The spec language for which to use is a bit wishy-washy...GL 3.2:
"In general, [equation 2.2] is used for signed normalized fixed-point
  parameters in GL commands, such as vertex attribute values."
"In general, [equation 2.3] is used for signed normalized fixed-point
  texture or floating point values."

ARB_vertex_type_2_10_10_10_rev clearly specifies equation 2.2.

*However*, OpenGL 4.2 and ES 3.0 both remove equation 2.2, instead 
saying to use equation 2.3 in all cases.

Intel hardware always uses equation 2.3, which provides a dilemma: 
should we just use the native hardware rules, which are fast, or do 
equation 2.2 via shader workarounds?

The maximum error/difference between the two equations is:

16-bit -> 0.00003
10-bit -> 0.00195
  8-bit -> 0.00784
  2-bit -> 0.66666

(Data and a script from Keith are available from:)
  http://people.freedesktop.org/~kwg/snorm/

Furthermore, since GL 4.2 is supposed to be backwards compatible, 
implementations can silently promote contexts to 4.2, likely giving them 
the new SNORM rules.  Which means applications can't really rely on the 
exact equation used...

Always using 2.3 seems preferable since it can actually represent 0.

I am tempted to rip out the code in vbo_attrib_tmp.h that implements 
equation 2.2 and always use 2.3.  That would also require changing 
Piglit's draw-vertices-2101010 test to accept either equation.

What do you all think?

Thanks for your time!
   --Ken