[Mesa-dev] [PATCH 8/9] nir: move to compiler

[Emil Velikov]

On 16 December 2015 at 20:24, Matt Turner <mattst88 at gmail.com> wrote:
> On Wed, Dec 16, 2015 at 8:53 AM, Emil Velikov <emil.l.velikov at gmail.com> wrote:
>> On 16 December 2015 at 01:13, Jason Ekstrand <jason at jlekstrand.net> wrote:
>>> On Sat, Nov 28, 2015 at 8:45 AM, Emil Velikov <emil.l.velikov at gmail.com> wrote:
>>>> On 27 November 2015 at 20:45, Jason Ekstrand <jason at jlekstrand.net> wrote:
>>>>> On Nov 27, 2015 11:26 AM, "Matt Turner" <mattst88 at gmail.com> wrote:
>>>>>> On Fri, Nov 27, 2015 at 6:50 AM, Emil Velikov <emil.l.velikov at gmail.com>
>>>>>> wrote:
>>>>>> > On 25 November 2015 at 22:01, Matt Turner <mattst88 at gmail.com> wrote:
>>>>>> >> On Wed, Nov 25, 2015 at 1:32 PM, Emil Velikov
>>>>>> >> <emil.l.velikov at gmail.com> wrote:
>>>>>> >
>>>>>> >>> --- a/src/Makefile.am
>>>>>> >>> +++ b/src/Makefile.am
>>>>>> >>> @@ -23,6 +23,7 @@ SUBDIRS = . gtest util mapi/glapi/gen mapi
>>>>>> >>>
>>>>>> >>>  # XXX: conditionally include
>>>>>> >>>  SUBDIRS += compiler
>>>>>> >>> +SUBDIRS += compiler/nir
>>>>>> >>
>>>>>> >> We have a non-recursive build in src/glsl today. I don't want to go
>>>>>> >> backwards.
>>>>>> > Not sure I fully get that can you elaborate ? Are you concerned that
>>>>>> > things won't build in parallel, increasing the compilation times ?
>>>>>> >
>>>>>> > On my dual core system running with -j2 results in approx 15 seconds
>>>>>> > increase. I'm willing to take that trade off for the improved
>>>>>> > readability. What is the difference on your system ?
>>>>>>
>>>>>> src/glsl has single Makefile that builds libglcpp, glcpp, libglsl,
>>>>>> glsl_compiler, glsl_test, libnir, and various test programs, allowing
>>>>>> all of these things to happen in parallel. The Makefile is perfectly
>>>>>> maintainable as it is and there's no advantage of splitting it,
>>>>>> especially when the work has been done to get things to this state
>>>>>> (commits 86d30dea, efd201ca) and NIR was added without an additional
>>>>>> Makefile.
>>>>>
>>>>> I would tend to agree.  Making things hierarchical is nice but,
>>>>> unfortunately, autotools makes this and parallelization mutually exclusive.
>>>>
>>>> Actually I have some ancient work where we benefit from both. Namely
>>>> have a single top level Makefile.am, which directly includes the
>>>> subdirectory Automake.mk files, resulting in one big Makefile at the
>>>> very end.
>>>>
>>>> That aside can we get some quantitative representation of the penalty
>>>> you guys see. On my (old) machine the difference is negligible  ~15
>>>> sec of a ~11 minute `make all' and ~16 minute `make distcheck'.
>>>
>>> What happened to this?  (Yes, "Busy doing a release" is a valid answer)
>>
>> There is a question in there for which I was waiting for an answer.
>> Here it is again - split from the rest, with a question mark (silly
>> me).
>>
>> Can anyone confirm how much of a penalty are we talking about (single
>> vs separate makefiles) ?
>
> I'm not going to take the time to quantify it. Just do a clean build
> and watch as 7 of your 8 cores sit idle as format_srgb.c is generated
> or shared-glapi/libglapi.la is linked. (A dual-core system is not
> going to demonstrate this issue properly)
>
>From a quick look at the logs (introducing these changes) I did not
see any numbers, thus the request.

> The point is that *I've already done the work* to prevent these kinds
> of things from happening in src/glsl and I am *not* interested in
> going back on that, especially for no reason at all.
>
I'm afraid I did not see the hint that you feel against (re)checking
things. If you can be a bit more blunt that'll be great.

> Just tell me if you don't want to do this project (moving things to
> src/compiler) and I'll take over.
>
Sure I'll re-spin things. Shame it'll shift away from the idea below.

>> Getting these makefiles benefit from parallelism, whist retaining the
>> split, is reasonably straight forward*.
>
> Unless there's something I've misunderstood or don't know... No, no it's not.
>
I've tried the following in a couple of places in gallium and it seems to work.

1) cd foo
2) sed -i "s|^\t|^\tfoo/|g" Makefile.sources
3) git mv Makefile.mk automake.mk
4) tweak automake.mk
4.1) append to (as opposed to set) variables like xx_LTLIBRARIES,
EXTRA_DIST, etc.
4.2) append foo/ or foo_ to binary/target names (don't forget about
include foo/Makefile.sources)
4.3) use per target C{XX}FLAGS as opposed to the AM_ ones
5) drop the makefile from configure.ac, change from SUBDIR = foo to
include foo/automake.mk in parent folder

* Note: one needs to "revert" 2 in the android/scons build. I've got
WIP for android but haven't tested it.

Above is inspired by openvswitch [1]

-Emil

[1] https://github.com/openvswitch/ovs