[Mesa-dev] Adding a SPIR-V back-end to the GLSL compiler

[Jason Ekstrand]

This is something that I have considered doing on multiple occasions and
have given a significant amount of thought.  Unfortunately, it has never
made it particularly high on my priority list so no code has been written.
However, there have been a number of people who have brought this up lately
so I thought I would do a brain-dump on the topic in the hopes of saving
someone some effort.

Why?

That's as good a place to start as any.  I think there a re a number of
reasons why this may be a useful thing to do:

 1) Provide another GLSL -> SPIR-V compiler.  Right now, the only thing in
that space is GLSLang and, while it seems to be servicing the need ok, it
also means that drivers tend not to implement SPIR-V support so much as
GLSLang support.  SPIR-V provides a decent amount of freedom in how you do
things and GLSLang tends to only do them one way.  This means that the
other 10 ways are untested in most drivers.  Having another SPIR-V producer
out there would help the ecosystem.

 2) Optimizations.  One thing that I've heard developers asking for is some
basic platform-agnostic optimizations specifically to reduce SPIR-V code
size.  Our compiler stack already has quite a few optimizations that
developers may want such as copy propagation, dead code elimination,
constant folding, loop unrolling, CSE, etc.  In a GLSL -> SPIR-V
translator, we would probably want these to be optional, but it wouldn't be
hard to provide them as flags.

 3) Bootstrapping GL_ARB_spirv.  The biggest barrier here is going to be
testing and convincing ourselves that we've done everything right.
However, if we did GLSL -> SPIR-V -> NIR -> back-end, then we would be able
to exercise both the GLSL -> SPIR-V path and the SPIR-V -> NIR path with
the full piglit suite.  Some people have suggested modifying
shader_runner.  However,  I think that if we simply did the SPIR-V
translation inside the driver, it would be much easier to test things like
non-SSO because we could go into SPIR-V after linking.

 4) Better testing spirv_to_nir.  See above.

 5) Better spec compliance.  I don't know the glslang code base well enough
to speak particularly strongly about it's correctness.  What I do know,
however, is that the GLSL compiler in mesa is used in production-grade
drivers that pass piglit, the Open GL 4.5 CTS, the Open GL ES 3.2 CTS, and
the Android CTS.  That's not something you can say about GLSLang.  I think
we should make that available to game devs wishing to target Vulkan.

Where?

This is a question that I have internally debated for some time.  When we
first started working on Vulkan, we boot-strapped it by adding some hacks
to the GLSL compiler to provide the descriptor set bindings and just used
GLSL.  When it came time for SPIR-V support, I chose to go directly into
NIR.  At the time, I thought that SPIR-V was going to end up being a fairly
low-level IR and that NIR would be a better match.  However, as things
turned out, SPIR-V (at least for graphics) is more of a binary form of GLSL
than anything else.  In retrospect, there are a number of things (dealing
with built-ins comes to mind) which would have been easier had we gone
SPIR-V -> GLSL.  However, I still stand by my original decision for two
reasons:

 1) As we try to add some of these "advanced compute" features that people
keep talking about, I think NIR will be a better and better fit.  Trying to
retrofit some of those things into GLSL may be more trouble than it's worth.

 2) Going directly into NIR meant that we didn't have to pull in mtypes.h.
NIR is a fairly stand-alone chunk of the mesa tree and we have pulled our
i965 back-end compiler out so it only depends on NIR.  This makes the
Vulkan driver much smaller and keeps it separate from the GL state tracker.

When considering a GLSL -> SPIR-V path there is some question about whether
it should be GLSL -> NIR -> SPIR-V or GLSL -> SPIR-V directly.  There are
pros and cons to both:

 1) SPIR-V would actually make a reasonable serialization format for NIR.
It's maybe not as compact as we could do if we made our own, but it's not
bad.

 2) NIR is, in some ways simpler than GLSL IR so it may be a bit easier.
Unfortunately, that's not always a help...

 3) Because NIR is simpler, it requires a lot of lowering.  There are
several things such as block variables (for UBOs and SSBOs) that NIR
doesn't know how to handle.  It assumes that it just gets load_ubo(index,
offset).  NIR also doesn't handle GLSL built-ins.  For SPIR-V -> NIR, those
things are lowered a way on-the-fly.  This means that we would have to add
some things to NIR and the SPIR-V code you get out would necessarily be
lower-level than what glslang produces.

 4) As I pointed out above, GLSL is a bit better fit than NIR in a lot of
ways.

 5) If we did NIR -> SPIR-V, we could also compile ARB programs to SPIR-V.
Maybe that's a downside?

All in all, my recommendation would be to do GLSL -> SPIR-V directly.  I
think there's less of an impedance mismatch there and I think it would be
easier to handle some of the higher-level things such as built-ins and
block variables.

How?

This is something that I've also given quite a bit of thought. :-)  You
could even say that I've written a decent chunk of it in my head.  Too bad
I haven't typed it into a computer yet. :-(

The first thing I would do would be to write some sort of SPIR-V builder
and put it in src/compiler/spirv.  It have multiple dword streams going at
a time (SPIR-V has multiple sections) to handle things such as types,
constants, etc. as well as function bodies.  Then you would have functions
which would generate SPIR-V opcodes similar to nir_builder.h.  For Types,
you would have a single spv_type function that turned a glsl_type pointer
into a SPIR-V ID.  You would want to have this backed by a hash map so that
they aren't re-emitted all the time.  For the rest of the opcodes,
SPIRV-Headers project on the Khronos github project has a JSON
representation of SPIR-V so it shouldn't be hard to write a little python
code that generates most of the "build an opcode" functions similar to what
we do for nir_builder_opcodes.h.

Finally, you would write the GLSL -> SPIR-V (or NIR -> SPIR-V) pass on top
of that.  If you did GLSL -> SPIR-V, you could probably look at glsl_to_nir
for inspiration.

Ok, there's my brain-dump.  It's on the list now so anyone who wants to
pick up the project has it.  I'm happy to chat further via e-mail or IRC
about different approaches but I think I've given a fairly complete picture
of my thoughts at the moment.  Happy Hacking!

--Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/mesa-dev/attachments/20170526/87f8a63a/attachment-0001.html>