[Mesa-dev] [PATCH] mesa: Use GLdouble for depthMax in final unpack conversions.

[Brian Paul]

On Mon, Aug 13, 2012 at 2:06 AM, Kenneth Graunke <kenneth at whitecape.org> wrote:
> The final step of _mesa_unpack_depth_span is to take the temporary
> GLfloat depth values and convert them to the desired format.  When
> converting to GL_UNSIGNED_INTEGER with depthMax > 0xffffff, we use
> double-precision math to avoid overflow and precision problems.
>
> Or at least that's the idea.  Unfortunately
>
>    GLdouble z = depthValues[i] * (GLfloat) depthMax;
>
> actually causes single-precision multiplication, since both operands are
> GLfloats.  Casting depthMax to GLdouble causes the scaling to be done
> with double-precision math.
>
> Fixes a regression in oglconform's depth-stencil basic.read.ds test
> since c60ac7b17993d28af65b04f9bbbf3ee74c35358c, where the expected and
> actual values differed slightly.  For example, 0xcfa7a6 vs. 0xcfa7a4.
>
> Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=49772
> Signed-off-by: Kenneth Graunke <kenneth at whitecape.org>
> ---
>  src/mesa/main/pack.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/src/mesa/main/pack.c b/src/mesa/main/pack.c
> index 83192c1..672a467 100644
> --- a/src/mesa/main/pack.c
> +++ b/src/mesa/main/pack.c
> @@ -4913,7 +4913,7 @@ _mesa_unpack_depth_span( struct gl_context *ctx, GLuint n,
>        else {
>           /* need to use double precision to prevent overflow problems */
>           for (i = 0; i < n; i++) {
> -            GLdouble z = depthValues[i] * (GLfloat) depthMax;
> +            GLdouble z = depthValues[i] * (GLdouble) depthMax;
>              if (z >= (GLdouble) 0xffffffff)
>                 zValues[i] = 0xffffffff;
>              else

Reviewed-by: Brian Paul <brianp at vmware.com>