1.6Gb of workdir/.../Dep - thoughts ...

[Michael Stahl]

On 01/06/12 11:16, Michael Meeks wrote:
> 
> 	And some thoughts on improving this; currently we add a ton of deps for
> packages that are really internal and typically change en-masse or not
> at all.
> 
> 	eg. 'boost' - it is installed, and then ~never changes again - people
> don't edit a single boost header and expect a dependency clean
> re-compile.
> 
> 	Ergo:
> 
> find -type f | xargs grep 'solver.*boost/' | wc -l
> 
> 	4.5 million hits (of 21.5million - worth having).

wow, that's a lot!

> 	IMHO we could - without significant loss of functionality reduce all
> those deps to a single stamp file (which we prolly install anyway) in
> the solver. Should be ~trivial to elide in our dep-re-writing anyway,
> save > 300Mb of space etc.
> 
> 	Should just be a few lines in solenv/bin/concat-deps.c

yes. just need to find the proper regex that doesn't have false positives :)

> 	My question would be: do we want to continue to make 'over precise
> deps' at this large computational, space and build performance cost
> possible ?

this sounds like a good idea to me, the external stuff tends not to
change very often, and when it does just re-building everything that
depends on it in any way is an acceptable cost; ccache will hopefully
take care of the cases where the change really doesn't affect the cxx file.

in a case like boost, which makes up a significant share of includes,
this would of course also make the dependency parsing of make faster,
which is, as you have found out, the bottleneck in make startup performance.

also, there are some external modules that don't deliver headers
properly or something, i remember the recent ICU upgrade breaking all
kinds of stuff in an incremental build because the header timestamps
were somehow wrong, so please do it for icu headers as well :)