[Piglit] Atomic Writes for piglit backends
baker.dylan.c at gmail.com
Tue Sep 23 17:55:47 PDT 2014
For a long time piglit has had a considerable problem with resuming
interrupted runs. This problem comes from the recovery tool, a very
fragile piece of code that relies on the formatting of the json to walk
backwards throwing away code. The problem with this approach are two
fold, first if the indentation changes the tool breaks. Second, it can
only handle corruption at the end of the file.
There is a second problem with our resume method, and that is that the
json file is read, truncated, and completely re-written on every resume,
this can be quite expensive.
Initially I began writing a complicated tool that actually could parse
json and clean it, but this quickly became a significant amount of code,
and the complexity was overwhelming, it was a complete project in
What I have implemented is a much simpler idea, that I borrowed from
databases, Atomic writes. By atomic we mean that a transaction should
either completely succeed or completely fail. Using a file backend there
are two basic ways to go about doing this. One method Copy the file to a
temporary location, write to the temporary file, verify the results, and
copy it back over the original. This has a couple of distinct drawbacks.
First it requires a lot of copying, second it means that the file needs
to be locked for long periods, and finally it requires reading the file.
What I implemented instead is a system that writes each test to it's own
file using built-in methods for both json and xml. At the end the runner
reads in each of these files, and creates a combined file from the
results. This has advantages and drawbacks. Because each test gets it's
own file there is no locking required at all, and flushing and syncing
is cheaper. It also means that there is no need to verify writes, when
the final file is being assembled json.load and xml.ElementTree.parse
are called on each file, and if they fail the result is considered
failed and ignored.
This series also takes a few more steps toward abstracting backends, by
creating a separate package for backends, and including a small subset
of the classes and functions from the package in the __init__ module,
allowing them to be accessed as if backends itself was a module. This is
sufficient for most applications, with mainly unit tests needing to
bypass this and access the modules directly.
Patch 1 removes a broken executable script that has been broken for some
Patch 2 does most of the abstraction
Patches 3-6 do some code cleanup
Patches 7 and 8 implement atomic writes in the two backends
Patch 9 does some additional refactoring that is possible becasue of
patches 7 and 8
More information about the Piglit