[Piglit] [Patch v2] framework: Update results to use versioned numbers
Ilia Mirkin
imirkin at alum.mit.edu
Sun Jun 8 09:54:09 PDT 2014
On Sun, Jun 8, 2014 at 12:12 PM, Ilia Mirkin <imirkin at alum.mit.edu> wrote:
> On Fri, Jun 6, 2014 at 5:02 PM, Dylan Baker <baker.dylan.c at gmail.com> wrote:
>> On Friday, June 06, 2014 04:57:28 PM Ilia Mirkin wrote:
>>> On Fri, Jun 6, 2014 at 4:54 PM, Dylan Baker <baker.dylan.c at gmail.com> wrote:
>>> > Ilia, I've tested this with your use case, does seem reasonable?
>>>
>>> The general idea seems reasonable, but I haven't had a chance to
>>> actually review or test myself. Do you have this in a branch somewhere
>>> so I can play with it? Or if you just want to push, we can fix up any
>>> issues later... whatever.
>>>
>>> > On Thursday, June 05, 2014 04:33:00 PM Dylan Baker wrote:
>>> >> This patch updates our json to version 1. Changes from version 0 to
>>> >> version 1 are as follows:
>>> >>
>>> >> - renamed 'main' to 'results.json'
>>> >> - dmesg must be a string (It was stored a list in some version 0
>>> >>
>>> >> results)
>>> >>
>>> >> - subtests are never stored as duplicate entries, a single instance of
>>> >>
>>> >> the test is recorded, (In version 0 both are possible)
>>> >>
>>> >> - there is no info entry in version 1, err, out, and returncode are
>>> >>
>>> >> always split into seperate entries
>>> >>
>>> >> This patch adds support to the results module for handling updates to
>>> >> the results in a sane way. It does this by adding a result_version
>>> >> attribute to the TestrunResult (which is stored as json), and
>>> >> implementing the ability to incrementally update results between
>>> >> versions.
>>> >>
>>> >> It does this automatically on load, non-destructively, moving the old
>>> >> results to results.json.old, but does write the updated results to disk,
>>> >> making the cost of this update a one time cost.
>>> >>
>>> >> v2: - Handle non-writable directories and files, this also fixes using
>>> >>
>>> >> file-descriptors as inputs
>>> >>
>>> >> Signed-off-by: Dylan Baker <baker.dylan.c at gmail.com>
>>> >> ---
>>> >
>>> > [snip]
>>
>> doh! I specifically pushed it to a branch and then forgot to put that in my
>> cover letter.
>> https://github.com/dcbaker/piglit versioned-results-v4
>
> Hm, just checked out that branch, this is what I get:
>
> $ python ~/src/piglit/piglit-summary-html.py -e pass -e skip
> nvc0-comparison <(xzcat nvcf-2014-04-20-mupuf.xz) <(xzcat
> nve7-2014-06-06-mupuf-gs5.xz) <(xzcat gk208-2014-06-07-mupuf-gs5.xz)
> --overwrite
> Traceback (most recent call last):
> File "/home/ilia/src/piglit/piglit-summary-html.py", line 26, in <module>
> from framework.programs.summary import html
> File "/home/ilia/src/piglit/framework/programs/summary.py", line 26,
> in <module>
> import framework.summary as summary
> File "/home/ilia/src/piglit/framework/summary.py", line 258, in <module>
> class Summary:
> File "/home/ilia/src/piglit/framework/summary.py", line 266, in Summary
> TEMPLATE_DIR = path.join(os.environ['PIGLIT_SOURCE_DIR'], 'templates')
> File "/usr/lib64/python2.7/UserDict.py", line 23, in __getitem__
> raise KeyError(key)
> KeyError: 'PIGLIT_SOURCE_DIR'
>
> Operator error of some sort? I tried rebuilding piglit in case there
> was some sort of compile-time-created dependency, but that didn't
> help.
Ugh, I guess that piglit top-level command is the only way to run
things now? I thought it was supposed to be optional... Still doesn't
work though...
$ python ~/src/piglit/piglit summary html -e pass -e skip
nvc0-comparison <(xzcat nvcf-2014-04-20-mupuf.xz) <(xzcat
nve7-2014-06-06-mupuf-gs5.xz) <(xzcat gk208-2014-06-07-mupuf-gs5.xz)
--overwrite
WARNING: Could not write updated results /dev/fd/63
WARNING: Could not write updated results /dev/fd/62
WARNING: Could not write updated results /dev/fd/61
Traceback (most recent call last):
File "/home/ilia/src/piglit/piglit", line 93, in <module>
main()
File "/home/ilia/src/piglit/piglit", line 88, in main
returncode = parsed.func(args)
File "/home/ilia/src/piglit/framework/programs/summary.py", line 98, in html
output.generate_html(args.summaryDir, args.exclude_details)
File "/home/ilia/src/piglit/framework/summary.py", line 429, in generate_html
path.join(destination, "index.css"))
File "/usr/lib64/python2.7/shutil.py", line 119, in copy
copyfile(src, dst)
File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile
with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory:
'/home/ilia/src/lib/piglit/templates/index.css'
I guess I have to run out of the piglit source dir directly otherwise
it assumes some sort of installation occurred? OK, running it out of
the source dir, the generated files seem a little wrong...
It generates a link to
nvcf-2014-04-20-mupuf/spec/EXT_framebuffer_object/fbo-blending-formats/3.html
but that file doesn't exist. It also seems to still be doubling
subtests, I suspect the issues are related. You can try it with those
same exact files yourself, they're in
http://people.freedesktop.org/~imirkin/piglit/. There's a "correct"
output for those 3 files available at
http://people.freedesktop.org/~imirkin/nvc0-comparison/problems.html .
Also it outputs
WARNING: Could not write updated results /dev/fd/63
WARNING: Could not write updated results /dev/fd/62
WARNING: Could not write updated results /dev/fd/61
Which is a little annoying... IMHO it shouldn't try to update the
results unless you ask for it, and with something like /dev/fd/*, it
doesn't have a chance.
Lastly, the time it takes to generate results has substantially
increased. With your branch:
$ time python ~/src/piglit/piglit summary html -e pass -e skip
~/piglit-results/nvc0-comparison <(xzcat
~/piglit-results/nvcf-2014-04-20-mupuf.xz) <(xzcat
~/piglit-results/nve7-2014-06-06-mupuf-gs5.xz) <(xzcat
~/piglit-results/gk208-2014-06-07-mupuf-gs5.xz) --overwrite
WARNING: Could not write updated results /dev/fd/63
WARNING: Could not write updated results /dev/fd/62
WARNING: Could not write updated results /dev/fd/61
real 0m56.297s
user 0m55.446s
sys 0m0.783s
With my local branch which has the fixups to deal with the subtests/info things:
$ time python ~/src/piglit/piglit-summary-html.py -e pass -e skip
nvc0-comparison <(xzcat nvcf-2014-04-20-mupuf.xz) <(xzcat
nve7-2014-06-06-mupuf-gs5.xz) <(xzcat gk208-2014-06-07-mupuf-gs5.xz)
--overwrite
real 0m6.800s
user 0m6.613s
sys 0m0.213s
IMO your branch isn't quite ready for prime-time.
-ilia
More information about the Piglit
mailing list