[Piglit] [Patch v2] framework: Update results to use versioned numbers

Sun Jun 8 17:55:15 PDT 2014

On Sun, Jun 8, 2014 at 8:47 PM, Dylan Baker <baker.dylan.c at gmail.com> wrote:
> On Sunday, June 08, 2014 12:54:09 PM Ilia Mirkin wrote:
>> On Sun, Jun 8, 2014 at 12:12 PM, Ilia Mirkin <imirkin at alum.mit.edu> wrote:
>> > On Fri, Jun 6, 2014 at 5:02 PM, Dylan Baker <baker.dylan.c at gmail.com>
> wrote:
>> >> On Friday, June 06, 2014 04:57:28 PM Ilia Mirkin wrote:
>> >>> On Fri, Jun 6, 2014 at 4:54 PM, Dylan Baker <baker.dylan.c at gmail.com>
> wrote:
>> >>> > Ilia, I've tested this with your use case, does seem reasonable?
>> >>>
>> >>> The general idea seems reasonable, but I haven't had a chance to
>> >>> actually review or test myself. Do you have this in a branch somewhere
>> >>> so I can play with it? Or if you just want to push, we can fix up any
>> >>> issues later... whatever.
>> >>>
>> >>> > On Thursday, June 05, 2014 04:33:00 PM Dylan Baker wrote:
>> >>> >> This patch updates our json to version 1. Changes from version 0 to
>> >>> >> version 1 are as follows:
>> >>> >>
>> >>> >> - renamed 'main' to 'results.json'
>> >>> >> - dmesg must be a string (It was stored a list in some version 0
>> >>> >>
>> >>> >>   results)
>> >>> >>
>> >>> >> - subtests are never stored as duplicate entries, a single instance
>> >>> >> of
>> >>> >>
>> >>> >>   the test is recorded, (In version 0 both are possible)
>> >>> >>
>> >>> >> - there is no info entry in version 1, err, out, and returncode are
>> >>> >>
>> >>> >>   always split into seperate entries
>> >>> >>
>> >>> >> This patch adds support to the results module for handling updates to
>> >>> >> the results in a sane way. It does this by adding a result_version
>> >>> >> attribute to the TestrunResult (which is stored as json), and
>> >>> >> implementing the ability to incrementally update results between
>> >>> >> versions.
>> >>> >>
>> >>> >> It does this automatically on load, non-destructively, moving the old
>> >>> >> results to results.json.old, but does write the updated results to
>> >>> >> disk,
>> >>> >> making the cost of this update a one time cost.
>> >>> >>
>> >>> >> v2: - Handle non-writable directories and files, this also fixes
>> >>> >> using
>> >>> >>
>> >>> >>       file-descriptors as inputs
>> >>> >>
>> >>> >> Signed-off-by: Dylan Baker <baker.dylan.c at gmail.com>
>> >>> >> ---
>> >>> >
>> >>> > [snip]
>> >>
>> >> doh! I specifically pushed it to a branch and then forgot to put that in
>> >> my
>> >> cover letter.
>> >> https://github.com/dcbaker/piglit versioned-results-v4
>> >
>> > Hm, just checked out that branch, this is what I get:
>> >
>> > $ python ~/src/piglit/piglit-summary-html.py -e pass -e skip
>> > nvc0-comparison <(xzcat nvcf-2014-04-20-mupuf.xz) <(xzcat
>> > nve7-2014-06-06-mupuf-gs5.xz) <(xzcat gk208-2014-06-07-mupuf-gs5.xz)
>> > --overwrite
>> >
>> > Traceback (most recent call last):
>> >   File "/home/ilia/src/piglit/piglit-summary-html.py", line 26, in
>> >   <module>
>> >
>> >     from framework.programs.summary import html
>> >
>> >   File "/home/ilia/src/piglit/framework/programs/summary.py", line 26,
>> >
>> > in <module>
>> >
>> >     import framework.summary as summary
>> >
>> >   File "/home/ilia/src/piglit/framework/summary.py", line 258, in <module>
>> >
>> >     class Summary:
>> >   File "/home/ilia/src/piglit/framework/summary.py", line 266, in Summary
>> >
>> >     TEMPLATE_DIR = path.join(os.environ['PIGLIT_SOURCE_DIR'], 'templates')
>> >
>> >   File "/usr/lib64/python2.7/UserDict.py", line 23, in __getitem__
>> >
>> >     raise KeyError(key)
>> >
>> > KeyError: 'PIGLIT_SOURCE_DIR'
>> >
>> > Operator error of some sort? I tried rebuilding piglit in case there
>> > was some sort of compile-time-created dependency, but that didn't
>> > help.
>>
>> Ugh, I guess that piglit top-level command is the only way to run
>> things now? I thought it was supposed to be optional... Still doesn't
>> work though...
>>
>> $ python ~/src/piglit/piglit summary html -e pass -e skip
>> nvc0-comparison <(xzcat nvcf-2014-04-20-mupuf.xz) <(xzcat
>> nve7-2014-06-06-mupuf-gs5.xz) <(xzcat gk208-2014-06-07-mupuf-gs5.xz)
>> --overwrite
>> WARNING: Could not write updated results /dev/fd/63
>> WARNING: Could not write updated results /dev/fd/62
>> WARNING: Could not write updated results /dev/fd/61
>> Traceback (most recent call last):
>>   File "/home/ilia/src/piglit/piglit", line 93, in <module>
>>     main()
>>   File "/home/ilia/src/piglit/piglit", line 88, in main
>>     returncode = parsed.func(args)
>>   File "/home/ilia/src/piglit/framework/programs/summary.py", line 98, in
>> html output.generate_html(args.summaryDir, args.exclude_details)
>>   File "/home/ilia/src/piglit/framework/summary.py", line 429, in
>> generate_html path.join(destination, "index.css"))
>>   File "/usr/lib64/python2.7/shutil.py", line 119, in copy
>>     copyfile(src, dst)
>>   File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile
>>     with open(src, 'rb') as fsrc:
>> IOError: [Errno 2] No such file or directory:
>> '/home/ilia/src/lib/piglit/templates/index.css'
>>
>> I guess I have to run out of the piglit source dir directly otherwise
>> it assumes some sort of installation occurred? OK, running it out of
>> the source dir, the generated files seem a little wrong...
>>
>> It generates a link to
>>
>> nvcf-2014-04-20-mupuf/spec/EXT_framebuffer_object/fbo-blending-formats/3.htm
>> l
>>
>> but that file doesn't exist. It also seems to still be doubling
>> subtests, I suspect the issues are related. You can try it with those
>> same exact files yourself, they're in
>> http://people.freedesktop.org/~imirkin/piglit/. There's a "correct"
>> output for those 3 files available at
>> http://people.freedesktop.org/~imirkin/nvc0-comparison/problems.html .
>>
>> Also it outputs
>>
>> WARNING: Could not write updated results /dev/fd/63
>> WARNING: Could not write updated results /dev/fd/62
>> WARNING: Could not write updated results /dev/fd/61
>>
>> Which is a little annoying... IMHO it shouldn't try to update the
>> results unless you ask for it, and with something like /dev/fd/*, it
>> doesn't have a chance.
>>
>> Lastly, the time it takes to generate results has substantially
>> increased. With your branch:
>>
>> $ time python ~/src/piglit/piglit summary html -e pass -e skip
>> ~/piglit-results/nvc0-comparison <(xzcat
>> ~/piglit-results/nvcf-2014-04-20-mupuf.xz) <(xzcat
>> ~/piglit-results/nve7-2014-06-06-mupuf-gs5.xz) <(xzcat
>> ~/piglit-results/gk208-2014-06-07-mupuf-gs5.xz) --overwrite
>> WARNING: Could not write updated results /dev/fd/63
>> WARNING: Could not write updated results /dev/fd/62
>> WARNING: Could not write updated results /dev/fd/61
>>
>> real    0m56.297s
>> user    0m55.446s
>> sys     0m0.783s
>>
>> With my local branch which has the fixups to deal with the subtests/info
>> things:
>>
>> $ time python ~/src/piglit/piglit-summary-html.py -e pass -e skip
>> nvc0-comparison <(xzcat nvcf-2014-04-20-mupuf.xz) <(xzcat
>> nve7-2014-06-06-mupuf-gs5.xz) <(xzcat gk208-2014-06-07-mupuf-gs5.xz)
>> --overwrite
>>
>> real    0m6.800s
>> user    0m6.613s
>> sys     0m0.213s
>>
>> IMO your branch isn't quite ready for prime-time.
>>
>>   -ilia
>
> Well, I designed it with the assumption that the conversion cost would be one
> time, so being a little slow wasn't a problem. Honestly, I've never run into
> anyone who uses that shell syntax, I had no idea it even existed.
>
> I think the copy bug is the same as the PIGLIT_SOURCE_DIR bug (which I have a
> fix for), and I'll see about enhancing the performance.

The performance hit might actually be due to the doubling that still
remains. I forgot to look, but I bet the number of tests was around
100K instead of 10K again for one of those runs. I remember when I was
first debugging the doubling issue, things ran very slowly as well.

  -ilia