[Mesa-dev] RFC: parallel shader compiles in mesa

[Connor Abbott]

There was already some work done for this (with i965) a while back:

http://lists.freedesktop.org/archives/mesa-dev/2014-July/063008.html

IIRC the reason it didn't get merged was that Chia never actually made
sure that the compiler was threadsafe (despite some patches fixing
some things). I can't seem to find any emails to confirm that, though.
Jason has mentioned that when he does parallel shader-db compiles on
his desktop, it sometimes crashes, so there are definitely still some
races left over. We've religiously avoided global variables and tied
everything to nir_shader and stack-local context variables in NIR for
exactly this reason, so it should be fine, but I'm not as sure about
GLSL IR (or for i965, the backends). Yes, it's painful work, but since
when is bolting on multithreading after-the-fact easy? (of ccourse,
any races are already a problem today, but I don't think people always
had that in mind when writing the code and this is going to make any
existing problems much worse).


On Sun, Nov 1, 2015 at 6:30 PM, Ilia Mirkin <imirkin at alum.mit.edu> wrote:
> AFAIK, nothing in the spec prevents shader compiles to happen in the
> background. Further, ARB_parallel_shader_compile provides a way to
> explicitly support this by providing a non-blocking query of current
> compile status. I take this to mean that any "regular" touch of the
> shader/programs will wait for the background compile to finish. Here's
> what I was thinking for implement this:
>
> I started by writing a threadpool with adjustable thread counts. I
> figure this will be shared by all contexts and default to the # of
> cores - 1, and be adjusted to the current context's # of threads
> whenever work is submitted.
>
> Both glCompileShader (-> gl_shader) and glLinkProgram (->
> gl_shader_program) will be made to compile asynchronously. Basically
> instead of running compile_shader and link_program directly, I'm going
> to submit them as work functions. Any lookup of the relevant objects
> will cause that lookup to wait for the relevant work to complete
> (except for the GL_COMPLETION_STATUS_ARB query). This will be enforced
> by the lookup_shader(_program)(_err) functions, and an additional
> helper will be made to wrap direct accesses.
>
> There's an additional problem that it'd be ideal if you could
> glCompileShader and then glLinkProgram without the glLinkProgram call
> waiting for the glCompileShader call to complete. I'm not keen on
> creating some complex dependency system for jobs, so I was thinking of
> something simple -- a predicate that determines whether work is
> doable, which is run before doing the work. If it's not, it's put into
> a separate list. Whenever any job completes, that side list is
> re-added to the main work list again. This will cause a little bit of
> extra work to be done, but I don't think it's too bad.
>
> I'm sure there will be a lot of specific issues that I'll have to
> tackle in making compile_shader/link_program asynchronous, but they
> seem to be mostly self-contained already. Does this overall approach
> sound reasonable?
>
> Thanks,
>
>   -ilia
> _______________________________________________
> mesa-dev mailing list
> mesa-dev at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/mesa-dev