[PATCH util/macros 1/2] Create CHANGELOG/INSTALL_CMD .tmp files in builddir

[Emil Velikov]

Hi Gaetan,

Thanks for the extensive answer. I hope with my replies below things
should be much clearer.

On 30 January 2017 at 15:18, Gaetan Nadon <memsize at videotron.ca> wrote:

> I no longer have a development environment so I cannot try anything.
>
> Maybe I missed the core idea of the patches. The srcdir is read only,
> correct?
>
s/is read-only/is read-only outside of the initial make dist/

To elaborate a bit - afaict make distcheck does the following
 - initial make dist
 - extract tarball, and run make all, make check, make dist (I'll
refer to this secondary one) et al.

> Unless I remember it incorrectly, the INSTALL and ChangeLog files are
> generated in the srcdir as they would normally be checked-in in git.
> Automake expects them to be there. It is the Xorg project who decided to
> generate them, not Automake. This situation is not "normal" to begin with.
>
Agreed.

> I assume you tried to build a git repo (AKA module) with the source code
> sitting on a read-only filesystem. Correct?
>
Not quite - I've reworked (simplified?) release.sh which pointed out
the (ab)use here.

> What are the reported failures?  Did the build stop because an O/S error? Or
> was it just that both files did not get generated.
>
In a gist "make distcheck failed" - upon the second `make dist' there
was an IO error.


Note: the INSTALL case is extra picky, so let's assume that all cases
below are dealing with ChangeLog.

> Currently: the INSTALL_CMD copies the INSTALL file from the datadir to
> top_srcdir/.INSTALL.tmp which is RO. That should fail on a RO filesystem,
> correct? IS that the problem you are trying to solve?
>
Precisely.

> Proposed patch: the INSTALL_CMD copies the INSTALL file from the datadir to
> top_builddir/.INSTALL.tmp/ which is RW.  That should work. A .INSTALL.tmp
> file now sits among the object code created by the compiler.
>
Correct.

> Next the top_builddir/.INSTALL.tmp is moved to the top_srcdir/INSTALL on a
> filesystem which is RO. That should equally fail, correct?
>
> I am assuming that if one cannot copy a file onto a RO filesystem, then one
> cannot move a file in there as well. Correct?
>
This part will succeed during the initial "make dist". The secondary
one has srcdir as RO so we'll end up in the "else" statement - rm
.FOO.tmp; touch FOO.

Note that if we reach the "else", we're
a) RO/RW file system and the file is present - touch is a
no-op/succeeds, respectively or
b) RW file system and the file is missing (building from a tarball and
manually removed the file) - touch will create a dummy one.

> On another topic, any change in the macros affect all modules (or git repos)
> in the X project. Extensive testing is required. The fix must also be
> backwards compatible with existing tarballs floating in the field.To get
> started quickly, try a few modules that have generated code and
> documentation. These are good test cases. As I remember it, all modules
> listed in build.sh built successfully with make dist.
>
I might be overly confident, but barring crazy hacks elsewhere things
should work across the board.
Then again, I was expecting xorg-macros to be missing blatant bugs so
I'm updating the script and pulling $world just in case.

Related questions:
Do you know of a script/semi-automated way to create a modfile to feed
into release.sh ?
I'm likely blind, but it seems that build.sh does not have an option
to only pull/clone the repos, without building, does it ? I would be
nice to do so and use release.sh which already does enough build/etc.
checking/testing.

Looking at build.sh I ended up adding 90 missing repos with fonts,
uitils, docs, others still to check/add. Not sure if/how much are
supposed to be legacy/deprecated but we shall see - if the newly added
ones start failing I'll just pull them out.

> A more general question, would a module generating source using lex and yacc
> would build on a RO filesystem?
>
> The requirement here is that the generated code requires lex and yacc
> programs to generate some C code when the repo is "configured" (autogen.sh).
> That generated code is then shipped in a tarball where the tarball builder
> only compiles the C generated code without the need to have the lex and yacc
> programs installed. A tarball builder may not run autogen.sh as this is a
> module configuration step and not a build step.
>
IMHO [definitely _not_ an expert wrt generated files] these ought to
be handled like INSTALL/ChangeLog - generate once at initial [non RO]
make dist, etc. Such that building from tarball is OK, even if one
does make clean.

Tl;Dr; Fully agree.

<thinking out loud>
IIRC in mesa we generate those files in builddir and think that we
purge them in make clean stage.
It's been on my list of things to check/fix but other distractions
keep on coming up ;-)
</thinking out loud>

> If your organization has a build strategy to use shared shared RO
> filesystems, this may not work for some modules. I had never seen this tried
> before. The INSTALL and ChangeLog files are not the only files created in
> the srcdir during the configuration step.
>
> An alternative would be to "configure" (autogen.sh) the module (without
> running make) on a RW filesystem and then move the code to a shared RO
> filesystem where it would be just "built" (make all).
>
Not sure what you mean with "your organisation", but let's leave this
out for a moment.

Afaict being able to do `git clone ...foo && cd foo && ./autogen.sh &&
make distcheck' should just work, shouldn't it ?
That's what (barring a unrelated sidestep) my earlier changes to
release.sh did, which in turn pointed to the issues here.

I think that you'll be able to quickly see the issue in question, if
you give the above one line (git clone..) a try. Atm it seems that
you're making quite nice, yet not strictly unrelated assumptions. Can
I bother you with that one, please ?

In reminds me of our first experience, where I came across another
funny build-related predicament are your deduction seemed, ahem, way
off [1] [2]. Although admittedly my wording was ... short, to put it
mildly.

Thanks
Emil

[1] https://lists.freedesktop.org/archives/xorg-devel/2012-October/034131.html
[2] https://cgit.freedesktop.org/xorg/xserver/commit/?id=ea3b7db4b747f883ef7b8b524ee13bf8a63add14