[Piglit] Atomic Writes for piglit backends

[Dylan Baker]

I always forget,

This is available at my github:
https://github.com/dcbaker/piglit.git submit/atomic-writes 

On Tuesday, September 23, 2014 05:55:47 PM Dylan Baker wrote:
> For a long time piglit has had a considerable problem with resuming
> interrupted runs. This problem comes from the recovery tool, a very
> fragile piece of code that relies on the formatting of the json to walk
> backwards throwing away code. The problem with this approach are two
> fold, first if the indentation changes the tool breaks. Second, it can
> only handle corruption at the end of the file.
> 
> There is a second problem with our resume method, and that is that the
> json file is read, truncated, and completely re-written on every resume,
> this can be quite expensive.
> 
> Initially I began writing a complicated tool that actually could parse
> json and clean it, but this quickly became a significant amount of code,
> and the complexity was overwhelming, it was a complete project in
> itself.
> 
> What I have implemented is a much simpler idea, that I borrowed from
> databases, Atomic writes. By atomic we mean that a transaction should
> either completely succeed or completely fail. Using a file backend there
> are two basic ways to go about doing this. One method Copy the file to a
> temporary location, write to the temporary file, verify the results, and
> copy it back over the original. This has a couple of distinct drawbacks.
> First it requires a lot of copying, second it means that the file needs
> to be locked for long periods, and finally it requires reading the file.
> 
> What I implemented instead is a system that writes each test to it's own
> file using built-in methods for both json and xml. At the end the runner
> reads in each of these files, and creates a combined file from the
> results. This has advantages and drawbacks. Because each test gets it's
> own file there is no locking required at all, and flushing and syncing
> is cheaper. It also means that there is no need to verify writes, when
> the final file is being assembled json.load and xml.ElementTree.parse
> are called on each file, and if they fail the result is considered
> failed and ignored.
> 
> This series also takes a few more steps toward abstracting backends, by
> creating a separate package for backends, and including a small subset
> of the classes and functions from the package in the __init__ module,
> allowing them to be accessed as if backends itself was a module. This is
> sufficient for most applications, with mainly unit tests needing to
> bypass this and access the modules directly.
> 
> Patch 1 removes a broken executable script that has been broken for some
> 		time
> Patch 2 does most of the abstraction
> Patches 3-6 do some code cleanup
> Patches 7 and 8 implement atomic writes in the two backends
> Patch 9 does some additional refactoring that is possible becasue of
> 		patches 7 and 8
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.freedesktop.org/archives/piglit/attachments/20140923/79dfcd3b/attachment.sig>