archive / install slowness ...

[Bjoern Michaelsen]

On Mon, Mar 04, 2013 at 10:31:12AM +0000, Michael Meeks wrote:
> Hi guys,
> 
> 	Norbert pointed out to me that one of the slowest pieces of our install
> is the:
> 
> 	'checking files with ARCHIVE flags' ...
> 
> 	Phase in the make_installer.pl foo, which burns a ton of CPU time for
> no aparently good reason.
> 
> 	Last I looked it seemed that we were using the perl Zip functionality
> to list files in a .zip and/or to un-compress it to /tmp for some
> reason.

http://opengrok.libreoffice.org/xref/core/solenv/bin/modules/installer/archivefiles.pm#145

> 	I believe we use .zip files as an easy way to package up a chunk of
> source code directory - say containing a template structure, into a
> bundle; and then install it en-masse in a simple rule and (more to the
> point) a single scp2 entry :-)
> 
> 	With the new make infrastructure / changes to scp2 - it would be just
> wonderful if we could stop: zipping that stuff during the build, and
> then unzipping it to /tmp during some very slow phase of
> make_installer.pl - and then copying it into the install-set; and simply
> copy it straight in ;->
> 
> 	Hopefully that no longer busts some conceptual flow ?

So I had a quick look at the above perl-foo and a grep on ARCHIVE in scp2 -- it
seems this is mostly done for external stuff like dictionaries etc. -- where we
do not have insight into what those builds do. And we also do not _want_ to
have insight into that. Doing a 'whatever is inside this tarball is the product
of external project foo' isnt that stupid of an approach.
Still I bet there are lots of ways this can be improved:
- build less l10n on the tinderbox
- dont use compressed zip, but something like an uncompressed tar archive
- put a reference to the directory in $WORKDIR in scp2 instead of the tarring the
  external product

All of the above is left as an excerise to a reader that is less disgusted by
Perl as I am. ;)

Best,

Bjoern