[Mesa-dev] [PATCH] mesa: sanitize input stencil mask values

[Haixia Shi]

Since the maximum supported precision for stencil buffers is 8 bits, we
should set bits 8:31 in the input mask values to zero.

The problem is that dEQP tests intentionally call glStencilFunc() with mask
values of ~0U (0xffffffffU) which overflows to -1 when converted to signed
integer by glGet* APIs.

Fixes 6 dEQP failing tests:
* dEQP-GLES3.functional.state_query.integers.stencil_value_mask_getfloat
* dEQP-GLES3.functional.state_query.integers.stencil_back_value_mask_getfloat
* dEQP-GLES3.functional.state_query.integers.stencil_value_mask_separate_getfloat
* dEQP-GLES3.functional.state_query.integers.stencil_value_mask_separate_both_getfloat
* dEQP-GLES3.functional.state_query.integers.stencil_back_value_mask_separate_getfloat
* dEQP-GLES3.functional.state_query.integers.stencil_back_value_mask_separate_both_getfloat

Signed-off-by: Haixia Shi <hshi at chromium.org>
Cc: Ian Romanick <ian.d.romanick at intel.com>
---
 src/mesa/main/stencil.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/src/mesa/main/stencil.c b/src/mesa/main/stencil.c
index b303bb7..f29187c 100644
--- a/src/mesa/main/stencil.c
+++ b/src/mesa/main/stencil.c
@@ -198,6 +198,11 @@ _mesa_StencilFunc( GLenum func, GLint ref, GLuint mask )
       return;
    }
 
+   /* Since the maximum supported precision for stencil buffers is 8 bits,
+    * set bits 8:31 in the input mask value to zero.
+    */
+   mask &= 0xFF;
+
    if (face != 0) {
       if (ctx->Stencil.Function[face] == func &&
           ctx->Stencil.ValueMask[face] == mask &&
@@ -258,6 +263,11 @@ _mesa_StencilMask( GLuint mask )
    if (MESA_VERBOSE & VERBOSE_API)
       _mesa_debug(ctx, "glStencilMask()\n");
 
+   /* Since the maximum supported precision for stencil buffers is 8 bits,
+    * set bits 8:31 in the input mask value to zero.
+    */
+   mask &= 0xFF;
+
    if (face != 0) {
       /* Only modify the EXT_stencil_two_side back-face state.
        */
@@ -468,6 +478,11 @@ _mesa_StencilFuncSeparate(GLenum face, GLenum func, GLint ref, GLuint mask)
 
    FLUSH_VERTICES(ctx, _NEW_STENCIL);
 
+   /* Since the maximum supported precision for stencil buffers is 8 bits,
+    * set bits 8:31 in the input mask value to zero.
+    */
+   mask &= 0xFF;
+
    if (face != GL_BACK) {
       /* set front */
       ctx->Stencil.Function[0] = func;
@@ -502,6 +517,11 @@ _mesa_StencilMaskSeparate(GLenum face, GLuint mask)
 
    FLUSH_VERTICES(ctx, _NEW_STENCIL);
 
+   /* Since the maximum supported precision for stencil buffers is 8 bits,
+    * set bits 8:31 in the input mask value to zero.
+    */
+   mask &= 0xFF;
+
    if (face != GL_BACK) {
       ctx->Stencil.WriteMask[0] = mask;
    }
-- 
2.8.0.rc3.226.g39d4020