[Mesa-dev] [PATCH 8/9] nir: move to compiler

[Jason Ekstrand]

On Wed, Dec 16, 2015 at 12:51 PM, Emil Velikov <emil.l.velikov at gmail.com> wrote:
> On 16 December 2015 at 20:24, Matt Turner <mattst88 at gmail.com> wrote:
>> On Wed, Dec 16, 2015 at 8:53 AM, Emil Velikov <emil.l.velikov at gmail.com> wrote:
>>> On 16 December 2015 at 01:13, Jason Ekstrand <jason at jlekstrand.net> wrote:
>>>> On Sat, Nov 28, 2015 at 8:45 AM, Emil Velikov <emil.l.velikov at gmail.com> wrote:
>>>>> On 27 November 2015 at 20:45, Jason Ekstrand <jason at jlekstrand.net> wrote:
>>>>>> On Nov 27, 2015 11:26 AM, "Matt Turner" <mattst88 at gmail.com> wrote:
>>>>>>> On Fri, Nov 27, 2015 at 6:50 AM, Emil Velikov <emil.l.velikov at gmail.com>
>>>>>>> wrote:
>>>>>>> > On 25 November 2015 at 22:01, Matt Turner <mattst88 at gmail.com> wrote:
>>>>>>> >> On Wed, Nov 25, 2015 at 1:32 PM, Emil Velikov
>>>>>>> >> <emil.l.velikov at gmail.com> wrote:
>>>>>>> >
>>>>>>> >>> --- a/src/Makefile.am
>>>>>>> >>> +++ b/src/Makefile.am
>>>>>>> >>> @@ -23,6 +23,7 @@ SUBDIRS = . gtest util mapi/glapi/gen mapi
>>>>>>> >>>
>>>>>>> >>>  # XXX: conditionally include
>>>>>>> >>>  SUBDIRS += compiler
>>>>>>> >>> +SUBDIRS += compiler/nir
>>>>>>> >>
>>>>>>> >> We have a non-recursive build in src/glsl today. I don't want to go
>>>>>>> >> backwards.
>>>>>>> > Not sure I fully get that can you elaborate ? Are you concerned that
>>>>>>> > things won't build in parallel, increasing the compilation times ?
>>>>>>> >
>>>>>>> > On my dual core system running with -j2 results in approx 15 seconds
>>>>>>> > increase. I'm willing to take that trade off for the improved
>>>>>>> > readability. What is the difference on your system ?
>>>>>>>
>>>>>>> src/glsl has single Makefile that builds libglcpp, glcpp, libglsl,
>>>>>>> glsl_compiler, glsl_test, libnir, and various test programs, allowing
>>>>>>> all of these things to happen in parallel. The Makefile is perfectly
>>>>>>> maintainable as it is and there's no advantage of splitting it,
>>>>>>> especially when the work has been done to get things to this state
>>>>>>> (commits 86d30dea, efd201ca) and NIR was added without an additional
>>>>>>> Makefile.
>>>>>>
>>>>>> I would tend to agree.  Making things hierarchical is nice but,
>>>>>> unfortunately, autotools makes this and parallelization mutually exclusive.
>>>>>
>>>>> Actually I have some ancient work where we benefit from both. Namely
>>>>> have a single top level Makefile.am, which directly includes the
>>>>> subdirectory Automake.mk files, resulting in one big Makefile at the
>>>>> very end.
>>>>>
>>>>> That aside can we get some quantitative representation of the penalty
>>>>> you guys see. On my (old) machine the difference is negligible  ~15
>>>>> sec of a ~11 minute `make all' and ~16 minute `make distcheck'.
>>>>
>>>> What happened to this?  (Yes, "Busy doing a release" is a valid answer)
>>>
>>> There is a question in there for which I was waiting for an answer.
>>> Here it is again - split from the rest, with a question mark (silly
>>> me).
>>>
>>> Can anyone confirm how much of a penalty are we talking about (single
>>> vs separate makefiles) ?
>>
>> I'm not going to take the time to quantify it. Just do a clean build
>> and watch as 7 of your 8 cores sit idle as format_srgb.c is generated
>> or shared-glapi/libglapi.la is linked. (A dual-core system is not
>> going to demonstrate this issue properly)
>>
> From a quick look at the logs (introducing these changes) I did not
> see any numbers, thus the request.
>
>> The point is that *I've already done the work* to prevent these kinds
>> of things from happening in src/glsl and I am *not* interested in
>> going back on that, especially for no reason at all.
>>
> I'm afraid I did not see the hint that you feel against (re)checking
> things. If you can be a bit more blunt that'll be great.
>
>> Just tell me if you don't want to do this project (moving things to
>> src/compiler) and I'll take over.
>>
> Sure I'll re-spin things. Shame it'll shift away from the idea below.

Thanks for working on this!  I'm hoping to start getting my SPIR-V
code into an upstreamable state soonish and I don't really want to put
it in src/glsl/nir/spirv.  That would be silly.

>>> Getting these makefiles benefit from parallelism, whist retaining the
>>> split, is reasonably straight forward*.
>>
>> Unless there's something I've misunderstood or don't know... No, no it's not.
>>
> I've tried the following in a couple of places in gallium and it seems to work.
>
> 1) cd foo
> 2) sed -i "s|^\t|^\tfoo/|g" Makefile.sources
> 3) git mv Makefile.mk automake.mk
> 4) tweak automake.mk
> 4.1) append to (as opposed to set) variables like xx_LTLIBRARIES,
> EXTRA_DIST, etc.
> 4.2) append foo/ or foo_ to binary/target names (don't forget about
> include foo/Makefile.sources)
> 4.3) use per target C{XX}FLAGS as opposed to the AM_ ones
> 5) drop the makefile from configure.ac, change from SUBDIR = foo to
> include foo/automake.mk in parent folder
>
> * Note: one needs to "revert" 2 in the android/scons build. I've got
> WIP for android but haven't tested it.

If we can make this work without killing scons, that would be awesome!
 (Personally, I'd be fine with killing scons, but I think others would
disagree. :-p)  I don't think anyone is against it.
--Jason