[Libreoffice-commits] 4 commits - bin/tinbuild2 bin/tinbuild_internals.sh README.tinbuild2
Libreoffice Gerrit user
logerrit at kemper.freedesktop.org
Fri Jan 11 01:57:27 PST 2013
README.tinbuild2 | 282 +++++++++++++++++++++++++++-------------------
bin/tinbuild2 | 28 ++++
bin/tinbuild_internals.sh | 6
3 files changed, 199 insertions(+), 117 deletions(-)
New commits:
commit 57b2c1cf905f327d8a81c730091ca65bbda49cda
Author: Norbert Thiebaud <nthiebaud at gmail.com>
Date: Fri Jan 11 03:56:54 2013 -0600
tb: major overall of the README
diff --git a/README.tinbuild2 b/README.tinbuild2
index f197f3f..c453d5e 100644
--- a/README.tinbuild2
+++ b/README.tinbuild2
@@ -8,105 +8,199 @@ build, or uploads the build results to http://dev-builds.libreoffice.org/daily/
Short version
-------------
+<profile_name> is a name you chose for you profile... usually use 'master' for a tinderbox
+that run the master branch. but you can call it anything you want.
+You can create different profile that do different kind of build, following the same procedure.
Read http://wiki.documentfoundation.org/Development/Tinderbox .
# clone tinbuild
-git clone git://anongit.freedesktop.org/libreoffice/contrib/buildbot ~/buildbot
+git clone git://gerrit.libreoffice.org/buildbot /lo/buildbot
# clone LibreOffice
-git clone git://anongit.freedesktop.org/libreoffice/core ~/master
+git clone git://gerrit.libreoffice.org/core /lo/core
# make the LibreOffice build
-cd ~/master
-./autogen.sh
+cd /lo/core
+./autogen.sh [ which ever argument are needed ]
make
make dev-install
# after the above succeeded, configure buildbot
mkdir -p ~/.tinbuild/config
-cat > ~/.tinbuild/config/your_name.cfg <<EOF
+mkdir -p ~/.tinbuild/metadata/<profile_name>
+cat > ~/.tinbuild/config/<profile_name>.cfg <<EOF
SMTPHOST=<your value>
SMTPUSER=<your value>
SMTPPW=<your value>
-TINDER_NAME=<your value>
-OWNER=<your value>
+TINDER_NAME=<your value> # [ see https://wiki.documentfoundation.org/Development/Tinderbox for naming conventions ]
+TINDER_ID=<id> # [ see https://wiki.documentfoundation.org/Development/Tinderbox for how to get an ID ]
+OWNER=<email address of the owner of this TB>
+METADATA_DIR=~/.tinbuild/meta/<profile_name> # this can be any existing directory you want
+ # if METADATA_DIR is not specified the tinderbox metadata are stored in the core repo itself
+GERRIT_PLATFORM=(MAC|LINUX|WINDOWS) # platform this TB is building when doing gerrit path verification
EOF
# save you autogen configuration.
mkdir -p ~/.tinbuild/autogen
-cp ~/master/autogen.lastrun ~/.tinbuild/autogen/your_name.autogen
+cp /lo/core/autogen.lastrun ~/.tinbuild/autogen/<prodile_name>.autogen
# check the configuration
-cd ~/master
-~/buildbot/bin/tinbuild2 -p your_name -c
+cd /lo/core
+/lo/buildbot/bin/tinbuild2 -p <profile_name> -c
# prime the tinderbox
-cd ~/master
-~/buildbot/bin/tinbuild2 -p your_name -z
+cd /lo/core
+/lo/buildbot/bin/tinbuild2 -p <profile_name> -z
# and if that succeeds, start the periodical building
-cd ~/master
-~/buildbot/bin/tinbuild2 -m all -p your_name
+cd /lo/core
+/lo/buildbot/bin/tinbuild2 -m all -p <profile_name>
-That's it; for more possibilities, like uploading the builds, or what to do
-when it complains that your architecture does not have flock, see the help,
-the below explanation, or the sources :-)
+# to also build gerrit patches
+/lo/buildbot/bin/tinbuild2 -m all -p <prodile_name> --mode=fair
-Longer explanation
-------------------
-Why:
-
-The main motivation was the need to have a more flexible customisation for the
-different build step. For performance reason, I needed to be able to set-up a
-ram_disk after the make clean, create a bunch of directory and link them in
-the tree to preempt solver and all the *.pro build directory
-
-I also needed a pre-clean step to be able to destroy that ram_disk (hence
-speeding up the make clean a lot)
-
-
-so rather than hacking the existing tinbuild in a way I could never upstream
-I started tinbuild2.
-
-while at it, I set-up a few more goals:
-
-Have the list of commits and committers - in case of failure - be generated
-based on the heads of the last sucessfull build, rather than based on a
-timestamp. this is needed because the timestamp associated to the commits
-have little correlation to the time when they have become available in the
-public tree.
-
-Working based on a set of heads, meant I also wanted to tinbuild to be
-'restartable'. in other words, if the tinbuild stop of any reason
-(Ctrl-C for instance). when it restart it will behave as if it was stopped
-just after the last sucessful build it did.
-This require to prime/bootstrap the tinbuild, that is to sucessfully complete a
-build sequence. this is achieved with runnning it once with -z.
-It will then attempt a build iteration (without pulling) and stop immediatly
-after that.
-
-There is a 'sanity-check' option which is intended to do reasonable attempt to
-verify that your configuration is workable. this check mostly for the existance
-of the config file and make sure that it result in the ability to send emails
-a test email is send to the owner declared in the config file.
-
-Due to the existance of build-time heisenbug, that seems related to
-bugs in the underlying OS/file-system/... I've added a mechanism to intercept
-these and prevent sending false negative reports. this is done by grepping
-the log produced by the build against a file containing a
-list of known heisenbug signature. if there is a match no email is sent
-and the build is retried immediatly on the same tree.
-The list of signature being highly machine dependant (and hopefully empty for
-most people), it is not checked in.
-That list live at ~/.tinderbuild/config/<project>.false_negatives
-This file is optional.
-
-The Uploading of dailies, using the -r <nnn> option can now be run
-asynchronously. That is the rsync job run in the background while
-tinderbox continue it's 'normal' operation.
+Configuration
+-------------
-How:
+The configuration of tinbuild2 is stored in ~/.tinbuild/*
+tinbuild2 is 'profile' based.
+each profile comport a config file in
+~/.tinbuild/config/<profile_name>.cfg
+and an assocaited autogen configuration for the build
+located at
+~/.tinbuild/autogen/<profile_name>.autogen
+
+optionnaly each profile can source an 'extension/override' script
+located at
+~/.tinbuild/phases/<profile_name>.sh
+that file, if it exsist will be sourced in tinbuild2 script.
+
+tinbuild2 maintain some metadata information in order to keep track ot the state
+of affair for a given profile.
+by default these files are sotre in the directory of the core repo, but
+if you specify a METADATA_DIR in the configuration of the profile then
+that directory will be used to store these files.
+Note: the METADATA_DIR, if specified, must exist.
+Using a METADATA_DIR allow to keep the repo clean, allow to use r/o repo,
+all to use git clean on the repo without risk of destroying the state of the tinderbox.
+
+config file
+-----------
+
+the config file for a profile is located at
+~/.tindbuild/config/<profile_name>.cfg
+
+The following parameter can od must be specified.
+
+Mandatory parameters:
+
+SMTPHOST : hostname of the smtp service to send email.
+SMTPUSER : user to pass to the smtp server to send email.
+SMTPPW : password for the smp service
+SMTPUSER and SMTPPW may be ommited if your smtp service connection without login. That is usually not the case.
+TINDER_NAME : name of yur tinderbox, please refer to
+ https://wiki.documentfoundation.org/Development/Tinderbox
+ for tinderbox naming conventions
+TINDER_ID : numerical id of your tinderbox. Please refer to
+ https://wiki.documentfoundation.org/Development/Tinderbox
+ for how to assign an id to your tinderbox
+OWNER : email address of the owner of this tinderbox.
+
+Optional parameters
+
+METADATA_DIR : directory where tinbuild2 will store the metadata information about the state of the tinderbox
+ if specified, the directory must exist (it is _not_ created by tinbuild2)
+ if this variable is not specified, then the metadata fiels are store in the root of
+ the core repository.
+ARTIFACTDIR : git repo where to deliver binary build.
+ tinbuild2 can collect binary build to form a bibisect git repo
+ this parameter is mandatody if you specifyc the '-x' command line argument
+ when running tinbuild2
+BIBISECT_GC : Y/N. In case of collection of binary artifact, make tinbuild2 to run a git gc command after
+ the collection of each new binary version
+BIBSECT_PUSH : Y/N. In case of collection of binary artifact, make tinbuild2 push the bibisect repo to
+ its remote 'origin' after the collection of each new binary version
+GERRIT_PLATFORM : MAC/LINUX/WINDOWS. when using tinbuild2 to do gerrit patch verification,
+ indicate which class of build that tinderbox build.
+ This parameter may be needed if the auto-termination that tinbuild2 do
+ using 'uname' is uncorrect, or possibly in the future to support cross-compile
+ tinderbox.
+PAUSE_SECONDS : <integer>. Duration, in seconds, to pause after each build, before trying to do another one.
+ The default value is 900 seconds (15 minutes).
+ For a dedicated tindebox machine, this should probably be 60 or less.
+
+command-line syntax:
+
+tinbuild2 -p <profile_name> [options]
+
+-0 Do incremental build; that is do not invoke the clean phase
+ between build. Not recommended for any mode of operation that involve
+ gerrit. In general that should be reserved to pure tinderbox mode
+ and that tinderbox need to be monitored closely to detect repeated failure
+ due to problem with incremental build.
+-a <email> for a run that do not report to the tinderbox server, but
+ is used to to a 'private' test of a particular branch -- like
+ a feature branch for instance -- this specify the email of
+ the 'author' to notify of the tinderbox activity.
+ this allow you to run a tinderbox to test a feature branch
+ and automatically send the result to a given person.
+ This work in conjonction with '-m author'. See below.
+-h print help.
+-c Execute a 'sanity check'. Verify that the configuration is sane
+ and send a test email to the owner, to verify that the smtp mechanism
+ works.
+-g <gerrit_ref> To a test build using the given gerrit ref to checkout a specific commit
+ A gerrit ref is in the form of : refs/changes/<n>/<m>/<p>
+ Message are sent in the Review section of the assocaited gerrit patch
+ to indicate that a build as started and to indicate the result of the build.
+ This option imply a one shot build. The tinbuild2 terminate after that one build.
+-i run with ionice -c3
+-k To not overrite autogen.lastrun if present. This allow to use -z
+ to do a quick test of the current config
+-m all|tb|owner|author|none Set the email level in regular tindeebox run
+ This does not apply to gerrit run or priming run.
+ all : send email to the tinderbox server, to the owner and to all the committer
+ involved in a failed build.
+ tb : send email to the tinderbox server and the owner
+ owner : send only tinderbox to the owner.
+ author : send email to the specified author (see -a above)
+ none : do not send email at all.
+--mode tb|gerrit|tb-gerrit|gerrit-tb|fair Mode of operation
+ tb : normal tinderbox mode. Wait on new commit to show on core
+ and build as new commits are found...
+ gerrit : gerrit patch verification mode. Query gerrit for patch to test-build
+ build them and report back, and repeat.
+ tb-gerrit: a mix of tb and gerrit, where if there is no new commit to be
+ build in tb mode, we attempt to get a task from gerrit
+ This mode favor the tb mode, in the sens that gerrit task
+ are only attempted if there is nothing left to do in tb mode
+ gerrit-tb : like tb-gerrit, except the the scheduling favor gerrit
+ in priority.
+ fair : dual mode tb and gerrit like tb-gerrit, except that the scheduling
+ priority is given alternatively to tb and gerrit.
+ The default mode is tb. --mode is not to be used conjunction with -z or -g
+ which are meant to trigger one-shot build.
+-n Run with 'nice'
+-r <bandwidth> Indicate that we want to push daily upload with a maximum upload
+ bandwidth as indicated in KB/s
+ This will attempt to upload a build at most once a day.
+ This require that you have obtained ssh keys to be able
+ to upload to dev-build.libreoffice.org
+-t Run the full suite of test after a sucessfull build
+-v Print more message while tinbuild2 is running
+-x Push sucessfull binary build, at most once a day to a bibisect repo
+ This require that ARTIFACTDIR be specified in the configuration of the
+ profile and point to a bibisect git repo.
+-z Run a 'Priming' build. In order for tinbuild2 to be able to notify
+ the appropriate subset of committers in case of failure, we need
+ a sucessfull 'reference' build.
+ You are required to run a sucessfull primer build before you can
+ run tinbuild2 in a loop.
+ This is required even if you do not actually email committers with failures.
+
+phases customization
+--------------------
The design is loosely inspired from Gentoo's portage. the idea is to use
the ability of bash to redefine function.
@@ -121,7 +215,7 @@ tb_call()
phase()
{
local f=${1}
- for x in {pre_,do_,_post}${f} ; do
+ for x in {pre_,do_,post_}${f} ; do
tb_call ${x}
done
}
@@ -148,50 +242,12 @@ reciprocally, every function should put a non-zero value in retval in case of
failure.
Now, the base function are implemented in tinbuild_phases.sh
-but these function can be overrided by a user-implementation.
+but these function can be overrided by a user-implementation,
+typically store in ~/.tinbphuid/phases/<profile_name>.sh
+which is sourced by tinbuild2 if it exists.
In the same spirit, platform dependent hack can be done in tinbuild_internals_<uname>.sh
The core default is meant to run on Linux, using bash and all the GNU tools that
are expected to normally be there on a Linux box.
-Other platform will re-implementent incompatible stuff in theyre own .sh
-
-And yes. bash is required. There maybe a way to avoid it, but:
-1/ Bash is required to build the product anyway
-2/ VHS won over BetaMax, get over it :-)
-
-
-Where:
-
-tinbuild2 rely on a profile name, passed with a mandatory -p <name> on the
-command line.
-That name is used to locate a file in ~/.tinbuild/config/<name>.cfg
-(you have to create it)
-
-That config file should contain all the needed SMTP information,
-the name of you build (TINDER_NAME) and
-can override factory default (like PAUSE_SECOND, which by default
-is set at 15 minutes (900 seconds))
-
-also, if present, the file ~/.tinbuild/phases/<name>.sh will be sourced
-after tinbuild_phases.sh.
-This allow to override and/or expand the default build behavior.
-
-Note that there is no 'inheritance'. if you override and existing function
-you cannot invoque it anymore. so you cannot 'expand' a function, just
-re-write you own implementation of it.
-
-Note that the file generated by tinderbuil2, typycally the capture of the output
-of the differents phase, is stored in file prefixed by tb_<project>_
-Some of them are 'rotated' to prev-* before starting a new cycle.
-
-This naming convention allow for the possibility to stop a run, checkout a different branch
-and run again using a different profile.
-In the future that will also allow for these step to be integrated in tinbuild2 to schedule
-and monitor multiple branch sequentially within one tinderbuild.
-
-Future:
+Other platform will re-implementent incompatible stuff in their own .sh
-* make one tinbuild able to handle more than one branch based on the same repo
-* make tinbuild respond to a pipe or a socket to be able to dynamically feed him
- one-time request to do a build on a feature-branch (in that case there won't
- be a baseline and the result, what-ever it is will be send to the 'requestor'
commit baeda4bbd126bdec7870554b6e53abe35b5ad41e
Author: Norbert Thiebaud <nthiebaud at gmail.com>
Date: Fri Jan 11 03:56:33 2013 -0600
tb2: syntax change in the gerrit plugin.
diff --git a/bin/tinbuild_internals.sh b/bin/tinbuild_internals.sh
index f11cdb4..b1f7526 100644
--- a/bin/tinbuild_internals.sh
+++ b/bin/tinbuild_internals.sh
@@ -463,7 +463,7 @@ check_for_gerrit()
GERRIT_TASK_TICKET=""
GERRIT_TASK_BRANCH=""
GERRIT_TASK_REF=""
- result=$(ssh ${GERRIT_HOST?} buildbot get-task -p core -a ${GERRIT_PLATFORM?} --format BASH)
+ result=$(ssh ${GERRIT_HOST?} buildbot get -p core --id ${TINDER_ID?} -a ${GERRIT_PLATFORM?} --format BASH)
[ $V ] && echo "get task result:${result}"
has_task=$(echo "$result" | grep "^GERRIT_TASK_")
@@ -682,10 +682,10 @@ local log_type="$1"
if [ "${retval?}" = "0" ] ; then
log_msgs "Repport Success for gerrit ref '$GERRIT_REF'."
- cat "${gzlog}" | ssh ${GERRIT_HOST?} buildbot report --ticket "${GERRIT_TASK_TICKET}" --succeed --log -
+ cat "${gzlog}" | ssh ${GERRIT_HOST?} buildbot put --id ${TINDER_ID?} --ticket "${GERRIT_TASK_TICKET}" --succeed --log -
else
log_msgs "Repport Failure for gerrit ref '$GERRIT_REF'."
- cat "${gzlog}" | ssh ${GERRIT_HOST?} buildbot report --ticket "${GERRIT_TASK_TICKET}" --failed --log -
+ cat "${gzlog}" | ssh ${GERRIT_HOST?} buildbot report --id ${TINDER_ID?} --ticket "${GERRIT_TASK_TICKET}" --failed --log -
fi
}
commit 3a55951b64bd3c3e8d81192bc593d5b1e5bf0d90
Author: Norbert Thiebaud <nthiebaud at gmail.com>
Date: Fri Jan 11 03:56:13 2013 -0600
tb2: in gerrit related mode of operation, make sure TINDER_ID is set
diff --git a/bin/tinbuild2 b/bin/tinbuild2
index 8583c8d..57d2a6c 100755
--- a/bin/tinbuild2
+++ b/bin/tinbuild2
@@ -440,6 +440,32 @@ if [ "${PUSH_TO_BIBISECT_REPO}" != "0" ] ; then
fi
fi
fi
+
+if [ "$MODE" = "gerrit-tb" -o "$MODE" = "tb-gerrit" -o "$MODE" = "gerrit" -o "$MODE" = "fair" ] ; then
+ if [ -z "$GERRIT_PLATFORM" ] ; then
+ os=$(uname)
+ case "$os" in
+ *Linux*)
+ GERRIT_PLATFORM="LINUX"
+ ;;
+ Darwin)
+ GERRIT_PLATFORM="MAC"
+ ;;
+ CYGWIN*)
+ GERRIT_PLATFORM="WINDOWS"
+ ;;
+ esac
+ fi
+ if [ -z "$GERRIT_PLATFORM" ] ; then
+ echo "Error: GERRIT_PLATFORM is required for mode involving gerrit"
+ exit 1
+ fi
+ if [ -z "$TINDER_ID" ] ; then
+ echo "Error: TINDER_ID is required for mode involving gerrit"
+ exit 1
+ fi
+fi
+
# if we want email to be sent, we must make sure that the required parameters are set in the profile (or in the environment)
case "$SEND_MAIL" in
all|tb|owner|debug|author)
commit d9bbd78b7c2d5dca47c712052f6356b065b1f2a8
Author: Norbert Thiebaud <nthiebaud at gmail.com>
Date: Fri Jan 11 03:55:29 2013 -0600
tinbuild2: sanity check relied on configure.in that is now configure.ac
diff --git a/bin/tinbuild2 b/bin/tinbuild2
index 309521b..8583c8d 100755
--- a/bin/tinbuild2
+++ b/bin/tinbuild2
@@ -157,7 +157,7 @@ sanity_check()
exit 1
fi
- if [ ! -f configure.in ] ; then
+ if [ ! -f configure.in -a ! -f configure.ac ] ; then
echo "Error: The current working directory must be the root git repo"
exit 1
fi
More information about the Libreoffice-commits
mailing list