[Mesa-dev] [PATCH] util: return 0 for NaNs in float_to_ubyte

[Brian Paul]

On 08/02/2018 09:36 PM, sroland at vmware.com wrote:
> From: Roland Scheidegger <sroland at vmware.com>
> 
> d3d10 requires NaNs to get converted to 0 for float->unorm conversions
> (and float->int etc.). GL spec probably doesn't care in general, but it
> would make sense to have reasonable behavior in any case imho - the
> old code was converting negative NaNs to 0, and positive NaNs to 255.
> (Note that using float comparison isn't actually all that much more
> effort in any case, at least with sse2 it's just float comparison
> (ucommiss) instead of int one - I converted the second comparison
> to float too simply because it saves the probably somewhat expensive
> transfer of the float from simd to int domain (with sse2 via stack),
> so the generated code actually has 2 less instructions, although float
> comparisons are more expensive than int ones.)
> ---
>   src/gallium/auxiliary/util/u_math.h | 11 +++++------
>   1 file changed, 5 insertions(+), 6 deletions(-)
> 
> diff --git a/src/gallium/auxiliary/util/u_math.h b/src/gallium/auxiliary/util/u_math.h
> index 79869a1..712305c 100644
> --- a/src/gallium/auxiliary/util/u_math.h
> +++ b/src/gallium/auxiliary/util/u_math.h
> @@ -360,7 +360,6 @@ uif(uint32_t ui)
>   
>   /**
>    * Convert ubyte to float in [0, 1].
> - * XXX a 256-entry lookup table would be slightly faster.
>    */
>   static inline float
>   ubyte_to_float(ubyte ub)
> @@ -375,16 +374,16 @@ ubyte_to_float(ubyte ub)
>   static inline ubyte
>   float_to_ubyte(float f)
>   {
> -   union fi tmp;
> -
> -   tmp.f = f;
> -   if (tmp.i < 0) {
> +   /* return 0 for NaN too */
> +   if (!(f > 0.0f)) {
>         return (ubyte) 0;
>      }
> -   else if (tmp.i >= 0x3f800000 /* 1.0f */) {
> +   else if (f >= 1.0f) {
>         return (ubyte) 255;
>      }
>      else {
> +      union fi tmp;
> +      tmp.f = f;
>         tmp.f = tmp.f * (255.0f/256.0f) + 32768.0f;
>         return (ubyte) tmp.i;
>      }
> 

Looks good.

Reviewed-by: Brian Paul <brianp at vmware.com>