[Mesa-dev] [PATCH] util: return 0 for NaNs in float_to_ubyte

[sroland at vmware.com]

From: Roland Scheidegger <sroland at vmware.com>

d3d10 requires NaNs to get converted to 0 for float->unorm conversions
(and float->int etc.). GL spec probably doesn't care in general, but it
would make sense to have reasonable behavior in any case imho - the
old code was converting negative NaNs to 0, and positive NaNs to 255.
(Note that using float comparison isn't actually all that much more
effort in any case, at least with sse2 it's just float comparison
(ucommiss) instead of int one - I converted the second comparison
to float too simply because it saves the probably somewhat expensive
transfer of the float from simd to int domain (with sse2 via stack),
so the generated code actually has 2 less instructions, although float
comparisons are more expensive than int ones.)
---
 src/gallium/auxiliary/util/u_math.h | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/src/gallium/auxiliary/util/u_math.h b/src/gallium/auxiliary/util/u_math.h
index 79869a1..712305c 100644
--- a/src/gallium/auxiliary/util/u_math.h
+++ b/src/gallium/auxiliary/util/u_math.h
@@ -360,7 +360,6 @@ uif(uint32_t ui)
 
 /**
  * Convert ubyte to float in [0, 1].
- * XXX a 256-entry lookup table would be slightly faster.
  */
 static inline float
 ubyte_to_float(ubyte ub)
@@ -375,16 +374,16 @@ ubyte_to_float(ubyte ub)
 static inline ubyte
 float_to_ubyte(float f)
 {
-   union fi tmp;
-
-   tmp.f = f;
-   if (tmp.i < 0) {
+   /* return 0 for NaN too */
+   if (!(f > 0.0f)) {
       return (ubyte) 0;
    }
-   else if (tmp.i >= 0x3f800000 /* 1.0f */) {
+   else if (f >= 1.0f) {
       return (ubyte) 255;
    }
    else {
+      union fi tmp;
+      tmp.f = f;
       tmp.f = tmp.f * (255.0f/256.0f) + 32768.0f;
       return (ubyte) tmp.i;
    }
-- 
2.7.4