archive / install slowness ...

[Michael Stahl]

On 04/03/13 12:36, Bjoern Michaelsen wrote:
> On Mon, Mar 04, 2013 at 10:31:12AM +0000, Michael Meeks wrote:
>> Hi guys,
>>
>> 	Norbert pointed out to me that one of the slowest pieces of our install
>> is the:
>>
>> 	'checking files with ARCHIVE flags' ...
>>
>> 	Phase in the make_installer.pl foo, which burns a ton of CPU time for
>> no aparently good reason.
>>
>> 	Last I looked it seemed that we were using the perl Zip functionality
>> to list files in a .zip and/or to un-compress it to /tmp for some
>> reason.
> 
> http://opengrok.libreoffice.org/xref/core/solenv/bin/modules/installer/archivefiles.pm#145
> 
>> 	I believe we use .zip files as an easy way to package up a chunk of
>> source code directory - say containing a template structure, into a
>> bundle; and then install it en-masse in a simple rule and (more to the
>> point) a single scp2 entry :-)

that's the problem.

>> 	With the new make infrastructure / changes to scp2 - it would be just
>> wonderful if we could stop: zipping that stuff during the build, and
>> then unzipping it to /tmp during some very slow phase of
>> make_installer.pl - and then copying it into the install-set; and simply
>> copy it straight in ;->
>>
>> 	Hopefully that no longer busts some conceptual flow ?
> 
> So I had a quick look at the above perl-foo and a grep on ARCHIVE in scp2 -- it
> seems this is mostly done for external stuff like dictionaries etc. -- where we
> do not have insight into what those builds do. And we also do not _want_ to
> have insight into that. Doing a 'whatever is inside this tarball is the product
> of external project foo' isnt that stupid of an approach.

actually that doesn't appear to be the case - none of the
dictionaries/Dictionary*.mk or */Zip*.mk use wildcard * to add files,
all list the files explicitly.

there are some zip files built by CustomTargets that just pack some
recursive directory though, e.g. the 2 different python zip files in
pyuno and python3.

i'd assume that in many cases zip files are entirely superfluous now;
they are a remnant from an age where partial builds of single modules
into a pre-built solver were a common workflow (and of course to avoid
duplication of lots of files in scp2).  we should be able to replace
almost all of them by converting them to Package*.mk files, and using
these directly to define the content of the installation set.

in the end it should look something like this:

1) instead of solver/ imagine some installation/ dir that is a runnable
   installation (LO and SDK and any help/l10n etc.)

2) */Package*.mk copies the files into the right place in installation/
   without any zipping or other nonsense

3) the RPM/DEB/archive/whatever instset creation basically works by
   taking the list of files from the */Package*.mk and putting the files
   into the proper RPM/... package
   (not sure whether it's best to reach this by writing some
   list-of-content file when building the Package, or reading all
   modules' Packages, or defining Packages more centrally...)

not quite sure what the best migration path to get there is; currently
most Package_*.mk just copy internal headers around which is quite
pointless in any case and probably should be fixed first.

then i guess we could start with a "list-of-contents" approach, and a
new "feature" in scp2 to replace ARCHIVE with this
(Package.mk-generated) list-of-contents file (i.e. gbuild writes the
list-of-contents file and scp2 reads it), and then incrementally migrate
things over.

or would it be easier to generate scp2's "output" files "directly" from
gbuild?  (i never looked at what is going on there anyway)