[Ezbench-dev] [PATCH 00/15] Make EzBench reboot-resistant when running piglit

[Martin Peres]

It seems like the time has finally come to address some shortcomings of
EzBench, namely handling gracefully reboots and crashes when using the
piglit runner!

To address this, I use the report's journal to detect unfinished runs. When
smartezbench is about to request a run, it first verifies if there are
unfinished runs that could be finished instead. When there are no unfinished
runs left, just create a new run.

The main work this series is addressing is actually related to the shell
runner. Indeed, core.sh was written 1.5 years ago and was meant as a user
interface. This led to very unfortunate choices which became limitations:

 - the number of rounds is the same for all the listed tests, which forces
   smartezbench to invoke core.sh once per test, leading to a lot of
   test parsing and environment setup;

 - querying information out of the runner was done using clumsy regular
   expressions which limited the amount of information smartezbench
   had access to.

 - timings were mostly learnt by core.sh but it was never very accurate for
   short tasks as there were many 

 - it is impossible to know what files got created, to avoid re-parsing the
   report entirely.

I mostly managed to work around the second issue, at the expense of the
complexity of core.sh. However, when trying to bolt on the resume support
in there, I was left with a choice: either rewrite a significant amount of
core.sh, or take the good parts out into a separate project (aptly named
runner.sh, which makes more sense than core.sh). I started researching the
second option and after a day of work, was left with a runner that was
way better-suited for machines and was much easier to hack on than core.sh!

The new runner os modeled on client-server model. When run, it takes no
parameters at all but instead listens to stdin commands (which makes it quite
easy to pipe commands in). It then outputs the commands it is running, adds
a lot of debug information (when wanted) and returns an error code and message
after every command, along with how long it took to execute (which will allow
smartezbench to know exactly how long it takes to deploy the environment,
parse the tests, etc...).

In this new runner, adding support for resume became quite trivial and most of
the code is in fact common between the 'run' and 'resume' command.

Adding resume support to piglit.test was done by splitting the report
generation out of __piglit_run__ so as it could be used by the newly-added
__piglit_resume__. I tried unifying again the IGT and piglit tests, but
again failed to do so in a maintainable way. At least, I managed to share the
report-generation!

Finally, I added a way to keep the generated report, as it may be useful for
other tools as well.

I hope you like the changes and I would love to hear your comments and reviews!

PS: These commits can be found in the runner.sh branch in the main repo.

Martin Peres (15):
  test/piglit: sort the output file to make diffing easier
  profile: introduce a helper to compare versions reliably
  profile/repos: add profile_repo_version_title
  profile/git-repo: fix profile_get_built_versions for the first run
  runner.sh: introduce a new leaner and more modular runner
  smartezbench: make use of the new runner
  report/journal: move the timestamp inside a attribute dict
  report/journal: parse the "test(ed)" ops differently
  journal: add the run's output file for the "test" and "tested" output
  report/journal: detect incomplete test runs
  test/piglit: allow resuming incomplete runs
  test/piglit: do not report "completed" if the report generation failed
  test/igt: add resume support
  smartezbench: resume incomplete runs, if possible
  test/piglit: allow keeping piglit/igt reports

 core.sh                                |   8 +-
 profiles.d/utils/auto-deploy.sh        |   4 +-
 profiles.d/utils/common.sh             |   7 +
 profiles.d/utils/git-repo.sh           |  14 +-
 profiles.d/utils/no-repo.sh            |   7 +
 python-modules/ezbench/report.py       |  51 ++-
 python-modules/ezbench/runner.py       | 179 ++++++++
 python-modules/ezbench/smartezbench.py | 157 ++++---
 runner.sh                              | 778 +++++++++++++++++++++++++++++++++
 tests.d/piglit/igt.test                |  81 ++--
 tests.d/piglit/piglit.test             |  42 +-
 unit_tests/ez_ut.py                    |   2 +-
 user_parameters.sh.sample              |   1 +
 13 files changed, 1198 insertions(+), 133 deletions(-)
 create mode 100755 runner.sh

-- 
2.11.0