[Mesa-dev] [RFC] NIR serialization

[Timothy Arceri]

On 12/09/17 14:23, Ian Romanick wrote:
> On 09/08/2017 01:59 AM, Kenneth Graunke wrote:
>> On Thursday, September 7, 2017 4:26:04 PM PDT Jordan Justen wrote:
>>> On 2017-09-06 14:12:41, Daniel Schürmann wrote:
>>>> Hello together!
>>>> Recently, we had a small discussion (off the list) about the NIR
>>>> serialization, which was previously discussed in [RFC] ARB_gl_spirv and
>>>> NIR backend for radeonsi.
>>>>
>>>> As this topic could be interesting to more people, I would like to
>>>> share, what was talked about so far (You might want to read from bottom up).
>>>>
>>>> TL;DR:
>>>> - NIR serialization is in demand for shader cache
>>>> - could be done either directly (NIR binary form) or via SPIR-V
>>>> - Ian et al. are working on GLSL IR -> SPIR-V transformation, which
>>>> could be adapted for a NIR -> SPIR-V pass
>>>> - in NIR representation, some type information is lost
>>>> - thus, a serialization via SPIR-V could NOT be a glslang alternative
>>>> (otoh, the GLSL IR->SPIR-V pass could), but only for spirv-opt (if the
>>>> output is valid SPIR-V)
>>>
>>> Ian,
>>>
>>> Tim was suggesting that we might look at serializing nir for the i965
>>> shader cache. Based on this email, it sounds like serialized nir would
>>> not be enough for the shader cache as some GLSL type info would be
>>> lost. It sounds like GLSL IR => SPIR-V would be good enough. Is that
>>> right?
>>>
>>> I don't think we have a strict requirement for the GLSL IR => SPIR-V
>>> path for GL 4.6, right? So, this is more of a 'nice-to-have'?
>>>
>>> I'm not sure we'd want to make i965 shader cache depend on a
>>> nice-to-have feature. (Unless we're pretty sure it'll be available
>>> soon.)
>>>
>>> But, it would be nice to not have to fallback to compiling the GLSL
>>> for i965 shader cache, so it would be worth waiting a little bit to be
>>> able to rely on a SPIR-V serialization of the GLSL IR.
>>>
>>> What do you suggest?
>>>
>>> -Jordan
>>
>> We shouldn't use SPIR-V for the shader cache.
>>
>> The compilation process for GLSL is: GLSL -> GLSL IR -> NIR -> i965 IRs.
>> Storing the content at one of those points, and later loading it and
>> resuming the normal compilation process from that point...that's totally
>> reasonable.
>>
>> Having a fallback for "some things in the cache but not all the variants
>> we needed" suddenly take a different compilation pipeline, i.e. SPIR-V
>> -> NIR -> ... seems risky.  It's a different compilation path that we
>> don't normally use.  And one you'd only hit in limited circumstances.
>> There's a lot of potential for really obscure bugs.
> 
> Since we're going to expose exactly that path for GL_ARB_spirv / OpenGL
> 4.6, we'd better make sure it works always.  Right?
> 
> One nice thing about SPIR-V is that all of the handling of uniform
> layouts, initial uniform values, attribute locations, etc. is already
> serialized.  If I'm not mistaken, that was one of the big pain points
> for all of the existing on-disk storage methods.  All of that has been
> sorted out for SPIR-V, and we have to make it work anyway.

Correct these are the main issues for the fallback path, however this is 
only used by i965 (exactly because an intermediate cache is missing).
Using SPIR-V as the intermediate cache means we still need to convert to 
NIR and run all the opts, so I don't really see the advantage of caching 
to SPIR-V over NIR.

Also there is going to be a requirement for a NIR cache for any of the 
Gallium nir based drivers (which possibly includes radeonsi in future).

> 
>> Serializing NIR, and possibly a few auxiliary structures that we need,
>> seems reasonable.  Although, just using the GLSL seemed reasonable to
>> me as well, but I guess that's proven to be painful?
>>
>> --Ken
>