[Mesa-dev] GLSL and alternative square matrix constructor, mat3x3 in Mesa

[randrianasulu at gmail.com]

Hello all!

Today i dived into GLSL for  first time, so sorry if i write something stupid.

One user at #radeon (nick <papillon81>) asked  why some shade  doesn't work.

Here is link to vertex shader as i saw it today:

http://mapserver.flightgear.org/git/gitweb.pl?p=fgdata;a=blob;f=Shaders/3dcloud.vert;h=135658ddb12c83e0652c85181f92575eeb06cf50;hb=HEAD

With unmodified Mesa master (after commit what enabled OpenGL 2.1/GLSL 1.20 
for r600 in master) there was error:
------------------
Compiling VERTEX source:
    1: // -*-C++-*-
    2: #version 120
    3:
    4: varying float fogFactor;
    5:
    6: attribute vec3 usrAttr1;
    7: attribute vec3 usrAttr2;
    8:
    9: float textureIndexX = usrAttr1.r;
   10: float textureIndexY = usrAttr1.g;
   11: float wScale = usrAttr1.b;
   12: float hScale = usrAttr2.r;
   13: float shade = usrAttr2.g;
   14: float cloud_height = usrAttr2.b;
   15:
   16: void main(void)
   17: {
   18:   gl_TexCoord[0] = gl_MultiTexCoord0 + vec4(textureIndexX, 
textureIndexY, 0.0, 0.0);
   19:   vec4 ep = gl_ModelViewMatrixInverse * vec4(0.0,0.0,0.0,1.0);
   20:   vec4 l  = gl_ModelViewMatrixInverse * vec4(0.0,0.0,1.0,1.0);
   21:   vec3 u = normalize(ep.xyz - l.xyz);
   22:
   23:   // Find a rotation matrix that rotates 1,0,0 into u. u, r and w are
   24:   // the columns of that matrix.
   25:   vec3 absu = abs(u);
   26:   vec3 r = normalize(vec3(-u.y, u.x, 0));
   27:   vec3 w = cross(u, r);
   28:
   29:   // Do the matrix multiplication by [ u r w pos]. Assume no
   30:   // scaling in the homogeneous component of pos.
   31:   gl_Position = vec4(0.0, 0.0, 0.0, 1.0);
   32:   gl_Position.xyz = gl_Vertex.x * u;
   33:   gl_Position.xyz += gl_Vertex.y * r * wScale;
   34:   gl_Position.xyz += gl_Vertex.z * w  * hScale;
   35:   gl_Position.xyz += gl_Color.xyz;
   36:
   37:   // Determine a lighting normal based on the vertex position from the
   38:   // center of the cloud, so that sprite on the opposite side of the 
cloud to the sun are darker.
   39:   float n = dot(normalize(-gl_LightSource[0].position.xyz),
   40:                 normalize(mat3x3(gl_ModelViewMatrix) * (- 
gl_Position.xyz)));;
   41:
   42:   // Determine the position - used for fog and shading calculations
   43:   vec3 ecPosition = vec3(gl_ModelViewMatrix * gl_Position);
   44:   float fogCoord = abs(ecPosition.z);
   45:   float fract = smoothstep(0.0, cloud_height, gl_Position.z + 
cloud_height);
   46:
   47:   // Final position of the sprite
   48:   gl_Position = gl_ModelViewProjectionMatrix * gl_Position;
   49:
   50: // Determine the shading of the sprite based on its vertical position 
and position relative to the sun.
   51:   n = min(smoothstep(-0.5, 0.0, n), fract);
   52: // Determine the shading based on a mixture from the backlight to the 
front
   53:   vec4 backlight = gl_LightSource[0].diffuse * shade;
   54:
   55:   gl_FrontColor = mix(backlight, gl_LightSource[0].diffuse, n);
   56:   gl_FrontColor += gl_FrontLightModelProduct.sceneColor;
   57:
   58:   // As we get within 100m of the sprite, it is faded out. Equally at 
large distances it also fades out.
   59:   gl_FrontColor.a = min(smoothstep(10.0, 100.0, fogCoord), 1 - 
smoothstep(15000.0, 20000.0, fogCoord));
   60:   gl_BackColor = gl_FrontColor;
   61:
   62:   // Fog doesn't affect clouds as much as other objects.
   63:   fogFactor = exp( -gl_Fog.density * fogCoord * 0.5);
   64:   fogFactor = clamp(fogFactor, 0.0, 1.0);
   65: }

VERTEX glCompileShader "" FAILED
VERTEX Shader "" infolog:
Error: mat3x3: undeclared function name.


Compiling FRAGMENT source:
    1: uniform sampler2D baseTexture;
    2: varying float fogFactor;
    3:
    4: void main(void)
    5: {
    6:       vec4 base = texture2D( baseTexture, gl_TexCoord[0].st);
    7:       vec4 finalColor = base * gl_Color;
    8:       gl_FragColor.rgb = mix(gl_Fog.color.rgb, finalColor.rgb, 
fogFactor );
    9:       gl_FragColor.a = mix(0.0, finalColor.a, fogFactor);
   10: }
   11:

Linking osg::Program "" id=29 contextID=0
Program's vertex attrib binding 10, usrAttr1
Program's vertex attrib binding 11, usrAttr2
glLinkProgram "" FAILED
Program "" infolog:
linking with uncompiled shader

---------------------------------

so, we dived in and found few links with some similar-looking problems.

Topic at Flightgear forum:
http://www.flightgear.org/forums/viewtopic.php?f=6&t=2387&p=77851

Message in Mac-OpenGL list:
http://lists.apple.com/archives/mac-opengl/2010/Aug/msg00000.html

vdrift developer's forum:
http://vdrift.net/Forum/viewtopic.php?t=734

And finally after modifying shader it compiles sucessfully:
we changed line 40
from 
normalize(mat3x3(gl_ModelViewMatrix) * (- gl_Position.xyz)));;
into 
normalize(vec3(gl_ModelViewMatrix * vec4(- gl_Position.xyz,0.0))));;

i tried to make simple patch, but it FAIL with real testing.

Patch was simple:
------------------------
diff --git a/src/mesa/slang/slang_typeinfo.c b/src/mesa/slang/slang_typeinfo.c
index d039a12..847e330 100644
--- a/src/mesa/slang/slang_typeinfo.c
+++ b/src/mesa/slang/slang_typeinfo.c
@@ -187,8 +187,11 @@ static const type_specifier_type_name 
type_specifier_type_names[] = {
    {"vec3", SLANG_SPEC_VEC3},
    {"vec4", SLANG_SPEC_VEC4},
    {"mat2", SLANG_SPEC_MAT2},
+   {"mat2x2", SLANG_SPEC_MAT2},
    {"mat3", SLANG_SPEC_MAT3},
+   {"mat3x3", SLANG_SPEC_MAT3},
    {"mat4", SLANG_SPEC_MAT4},
+   {"mat4x4", SLANG_SPEC_MAT4},
    {"mat2x3", SLANG_SPEC_MAT23},
    {"mat3x2", SLANG_SPEC_MAT32},
    {"mat2x4", SLANG_SPEC_MAT24},

as GLSL spec says on page 20:
 "mat3x3 same as a mat3" - 
http://www.opengl.org/registry/doc/GLSLangSpec.Full.1.20.8.pdf

But because my patch fail - i guess i need add  new type into slang_typeinfo.h 
and then modify
slang_typeinfo.c with completely new type (for other matNxN [square] cases 
too).

I understand all work now focused on glsl2 branch, but may be our work (if i 
produce anything working) will be useful for 7.8 ?