[Piglit] [RFC] ARB_gl_spirv: opening discussion for borrowing tests

Thu Apr 19 11:07:47 UTC 2018

Hi everybody,

recently I have sent to the list the first series with patches for
testing ARB_gl_spirv [1]. On the cover letter I mention "barebone tests"
and "borrowed tests", and the need fordiscussion, so here we are. And
sorry if the email is somewhat long and messy, but it is intended as a
brainstorm. FWIW, this was already introduced in my presentation at
FOSDEM, as some people (specifically Mark Janes) asked about that.

* Barebone tests
Basically smoke testing for all the general shader features supported
(uniforms, UBO, SSBO, etc), plus tests for some specific ARB_gl_spirv
features (like specialization constants). Why "just barebone"? The issue
with this extension is that the functionality it covers is basically all
current GLSL functionality, with some exceptions, so it is complex to
get a full testing coverage from scratch. So the idea would be to touch
a little of everything,  so any implementation of the extension should
pass, but would not provide a full coverage of the extension for
practical reasons. We would leavethat for the "borrowed tests" (more
about that below)

An additional advantage of this approach is that those GLSL tests are
compatible withthe ARB_gl_spirv/GL_KHR_vulkan_glsl and would not need an
automatic conversion like the borrowed tests from other specs (more
about that below).

The series I sent has25 tests. On our development branch we have ~70
individual tests, and we foresee that the final version would have a
little more than100 tests.

We also have some hundred generated builtin tests. Those generate
SPIR-Vdirectly. This allows testingthem without the need torely on the
borrowed tests, so it avoidsthe dependency on glslang to test them. It
also reduces the number of builtin tests needed for a SPIR-V run, as we
don't bother generating tests that doesn't make sense for SPIR-V, such
as operators with mixed types.

Thesebarebone tests only add a dependency to spirv-tools, as the SPIR-V
is included as assembly. And in any case, it is executed as anexternal
binary, as we assumed that a runtime dependency would be easier to be
accepted thana build dependency. But it would be easy to replace this
with a dependency onspirv-tools, and call the library directly. As far
as we know, adding a dependency to spirv-tools would be considered as
"fine".

Some people would argue that most of that is already tested on other
specs, so it would be redundant, and we should only test the
ARB_gl_spirv specifics, and let the full general coverage on the
borrowed tests. But there are some extra things to discuss about that.

* Borrowed tests

Those are the shader_tests that we re-use from other specifications. In
this case, the conversion to SPIR-V is a two pass process, done by a
custom script. In thefirst pass the script checks the shader and fixesit
in order to be compatible with glslang shader expectations, and then the
script calls glslang to convert it to SPIR-V.  Fix in this case means
things like change gl_Vertex/gl_FragColor to generic in/outs. In
addition to fixingthe shaders, the script hassome features like
markingthe tests to be skipped on the following piglit run, and allows
storingthe SPIR-V assembly in anexternal file, or include it in the
shader_testitself. The former is mostly useful for borrowed tests, the
latter for barebone tests.

Nicolai started that script, that we have been using and extended for a
while. Right now we have the current conversion rate:
   * piglit/tests directory: [2028] skip: 537, excluded: 0, success:
1363, fail: 128
   * piglit/generated_tests directory: [32354] skip: 1110, excluded: 0,
success: 31244, fail: 0
That is a (IMHO) good conversion rate right now.

Issues:
   * Dependency on glslang to create the binaries: right now the
GLSL->SPIR-V pass is done using glslang. As with spirv-tools, as a
runtime binary, but again we could use the library. The problem with
thisapproachis that after talking with some people, they don't consider
glslang as a "fine" dependency of piglit. The library API is somewhat
unstable (although not sure if that changed lately), and there are also
some issues with the SPIR-V generated now and then. So there is the
possiblity of this dependency to be vetoed. This is also the reason
barebone tests include the assembly, and only depend on spirv-tools to
re-assemble it. The idea is that for the barebone tests it should not
matter how SPIR-Vwas generated, you could do it writing SPIR-V assembly
directly without any tool, for example. FWIW, as part of our work we
created issues on glslang and provided some patches, and we got
feedback/fixes, so the library is active and receptive to suggestions.

   * Dependency on glslang to fix the shaders: this is something new
that didn't have Nicolai'soriginal script. Initially we added support
for fixing uniforms lacking explicit locationson the script directly.
But glslang has already some limited support for this, and other
automatic fixes, so we moved to a mixed approach of using glslang
features plus some minor fixes. The main advantage of this is that
reduces how much the script modifies GLSL, letting glslang to do that.
So if finally glslang is vetoed, we would need to reimplement those
features.

   * Not other real alternative right now: if glslang is vetoed, we
would need to face that right now there is no real alternative. There is
one coming, as Ian is working on a Mesa backend for SPIR-V [1], first
suggested on a previous thread [2]. Additionally, Ian's work adds an
interesting alternative. If Mesa itself is able to do the GLSL->SPIR-V
pass, then it would be feasible to add a special debug mode configured
by an envvar, so internally it would do the GLSL->SPIR-V pass, and then
the linking would use the ARB_gl_spirv path. In that way we would not
need a custom script on piglit, but to enablethis envvar. But then this
backend would need to also support shader fixing (more advanced
thanglslang), and this would not work for non-Mesa drivers using piglit
(although Iam not sure if this is a use case with high priority for
piglit). In any case, as I said, this alternative is still not available.

   * Slow conversion: assuming that glslang dependency is okish, ideally
we would like the GLSL->SPIR-V conversion to be done while building
piglit, as with the generated tests. But the problem is that it is a
slow process. Tests directory need 2min, and generated_tests need 50 min
on my i5. Those times are with one process, but although we added
support for forking N processes, generated_tests conversion would be
already too slow, 25 minutes on my i5.

   * Slow piglit run: with ARB_gl_spirv we would be duplicating the
tests to run on the shaders.py profile. And in practice, the time needed
would be more than the double

Due to all this issues, one alternative would betonot include/run by
default the SPIR-V borrowed tests, and only include the script and
instructions to usein piglit, so that any driver developer interested in
the feature can useit. But as we know, usually tools that are not used
tend to rot.

Taking into account the above considerations, our suggestion is the
followingcompromise:
  * Go on with the barebone tests. Add a spirv-tools runtime dependency
(those tests would just fail without spirv-tools)
  * Accept the use of glslang, as there is no real alternative right now.
  * Continueexecuting itas a binary, sothat it remainsa runtime
dependency. This way, platforms not implementing the
ARB_gl_spirvextensionwon't have adependency onglslang, and those tests
willsimplyfailor be skippped .
  * Only convert to SPIR-V the shaders of the test directory and not the
generated_tests
       * Not only because of the time to convert the generated_tests. 
       * Also because as mentioned, we are adding SPIR-V generated tests
for some of the existing cases.
       * Revisit if generated_tests got slimmed
  * Get the shader.py profile to do a new run but passing a --spirv
option to the test directory (or perhaps a new generated_spirv_tests
directory)
  * Revisit everything when the Mesa GLSL->SPIR-V backend got in good shape.

Opinions?

PS: if you are curious, here the comparison of a full piglit shader.py
run using GLSL vs using SPIR-V with our really-dirty development branch
for i965 on Skylake:

       name: 20180418-glsl-code-p 20180418-spirv-code-
       ----  -------------------- --------------------
       pass:                30475                28324
       fail:                   15                  287
      crash:                    5                   41
       skip:                 3887                 5730
    timeout:                    0                    0
       warn:                    0                    0
 incomplete:                    0                    0
 dmesg-warn:                    0                    0
 dmesg-fail:                    0                    0
    changes:                    0                 2176
      fixes:                    0                    9
regressions:                    0                  324
      total:                34382                34382

[1] https://lists.freedesktop.org/archives/piglit/2018-April/024145.html
[2]
https://lists.freedesktop.org/archives/mesa-dev/2017-November/176407.html
[3] https://lists.freedesktop.org/archives/mesa-dev/2017-May/156592.html
[4] https://fosdem.org/2018/schedule/event/spirv/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/piglit/attachments/20180419/9048da9e/attachment-0001.html>