[Mesa-dev] gl4_gears - SPIR-V and shader module reflection

[Michael Clark]

On 11/30/20 3:38 PM, Michael Clark wrote:
> Hi again,
> 
> On 11/24/20 9:43 PM, Michael Clark wrote:
>> - kitty_gears - OS Mesa gears using kitty terminal graphics protocol
>> - gl1_gears - OpenGL 1.x gears using GLFW similar to the original
>> - gl2_gears - OpenGL 2.x gears using VAO/VBO, programmable shaders
>> - vk1_gears - Vulkan 1.x gears using spirvtools and SPIR-V shaders
> 
> I decided to use this app to experiment with some of the new OpenGL 
> features in particular the ability to share shader modules between 
> OpenGL and Vulkan. Be nice because we will have portable binaries...
> 
> subsequently added:
> 
> - gl3_gears - moved vertex array objects here and downgraded gl2_gears
> - gl4_gears - added uniform buffer objects and SPIR-V shader modules
> 
> OpenGL 4.x SPIR-V attribute reflection branch is here:
> 
> - https://github.com/michaeljclark/glkitty/tree/opengl4-experiment
> 
> gl4_gears is decidedly experimental and there issues with attribute 
> reflection that need to be resolved. NVIDIA OpenGL 450 implementation 
> has bugs with reassignment of attribute locations when using the 
> #version 450 shader dialect with explicit location, used by the SPIR-V 
> shader modules, so a different set of locations is used in comparison to 
> vk1_gears. likewise Mesa has a somewhat alternate behavior and also 
> appears buggy. It's really a bit messy, especially given I still want to 
> perform reflection but now the shaders have mandatory location attrs. 
> that seems to me like a backwards step for hot reloading of experiments.
> 
> See the #define in: shaders/gears.vert#L3-L5
> 
> - GENERIC_VULKAN - set by default to enable vk1_gears
> - NVIDIA_OPENGL - set to run gl4_gears with nvidia-450
> - MESA_OPENGL - set to run gl4_gears with Mesa 20.0.8
> 
> similarly there is a USE_SPIRV define near the top of gl4_gears.c
> which currently has only been tested to work with NVIDIA OpenGL 450
> 
> need to spend some time reading ARB_gl_spirv.txt. it seems reflection is 
> becoming optional instead of intrinsic, which is a step backwards for 
> OpenGL, although I can see why OpenGL ES, of potentially large games, 
> may want to statically link offsets, but previously I could rely on 
> dynamic evaluation of shader module attribute names. It makes life easy 
> for the developer/user of these APIs.
> 
> noticing I might be able to compile separate SPIR-V modules:
> 
> #ifdef GL_SPIRV
> 
> which is also not ideal because now I don't have portable binaryies 
> given its silly (now manual) linkage. it seems reflection is optional 
> for SPIR-V, and the attribute reflection is not yet? working in Mesa? 
> also it seems if I find the attribute I can't relink it. there is a 
> whole set of new APIs to explore. just an fyi on a gl4_gears that could 
> be an interesting test cast for mesa-demos as this port of gears is 
> still pure C. I am actually working in C++, but tend to like to isolate 
> dependencies when working on small tests for context initialization and 
> what not...
> 
> anyway keep up the good work!!! i'm surprised how close Mesa is to 
> supporting all of the latest APIs. the next step was exploring bindless...

I managed to get a solution that works with NVIDIA Vulkan and OpenGL 
drivers as well as the Mesa Drivers in Ubuntu. I did this by disabling 
reflection and just scanning the re-assigned in a reverse lookup from 
the name, which is ironic. It seems that reassigning the indices does 
not work on some drivers. I was doing that because of NVIDIA driver's 
dislike for 0.

It begs the question of where OpenGL, OpenGL ES and GLSL are headed. I 
am quite interested in the position of Intel and freedesktop on the 
SPIR-V draft:

https://www.khronos.org/registry/OpenGL/extensions/ARB/ARB_gl_spirv.txt

location seems like redundant boiler plate to me. It seems natural to 
see the declarations like a struct and use declaration order if location 
is not present. This would not affect Vulkan in any substantive way, as 
it would be an "re-addition" versus a "subtraction" of reflection to the 
OpenGL suite of APIs. I am a proponent of concise and hot-reload.

Supposedly necessary but unnecessary complexity. Like an artist 
protecting his encrypted SPIR-V assets so they can make a crust on the 
web, instead of asm.js and text-only for certain assets. And of course 
(location = ?) does not make sense there. I understand the need. It just 
seems there are some mutually exclusive forces at play. Why can't I have 
a binary module and export the names of functions and struct members,

Vulkan would need some utility interfaces outside of the driver to find 
the strings in the SPIR-V, but it seems straight forward given the 
constraint has been "added", with both OpenGL and OpenCL traditionally 
being relatively low-level interfaces with introspection and reflection.

Just so you are aware of one independent developer's position. I would 
like to reflect on function arity and make JIT dispatch proxies so that 
C can transparently call GLSL code with fixed sized arrays. The GLSL is 
the safe subset of the language at the moment because it assumes the 
platform has made guarantees about the aliasing of various arrays. I 
know there is SYCL, but I  would actually like a GLSL-inspired script 
language that I can run server side with SPIR-V and C2x. GLSL is 
actually more portable with the pure function subset of C, than say Go. 
Hence my interest. I know there is SYCL, but I see templates as 
confusing the issue of layering at the linkage layer. I think the 
program object introspection APIs are essential, and it's with an OpenGL 
4 like introspection API that I would build a modern linker. With JIT 
and libffi, I could probably make it possible for compute shaders to 
call other compute shaders and C. On OS Mesa of course... An upcall for 
printf. I'm not sure how that is handled.

SPIR-V environment-call? and creation of interrupt linked stubs to 
interrupt the calling environment. printf would needs an extension for 
format strings. anonymous arrays with an integer id in the SPIR-V?

C-with-variable-length-arrays on SPIR-V headless target. Does it exist?

Regards,
Michael.