[Piglit] Fix subtests
Tom Stellard
tom at stellard.net
Tue Nov 12 14:26:35 PST 2013
Hi Dylan,
Overall these look pretty good, just a few issues:
'Not Run' -> pass changes are not showing up in the 'changes' page.
You can reproduce this with the attached result files by passing them
to piglit-summary-html.py in this order:
get-num-groups-exit-early get-num-groups-good get-num-groups-partial-fail
If I put get-num-groups-good first in the list, then the 'changes' page
displays the correct results.
Another minor issue is that the parent groups for subtests
displays 0/1 or 1/1 no matter how many subtests there are. It would be
nice if the number of subtests could be displayed (but I can live
without this).
Thanks for working on this.
-Tom
On Mon, Nov 11, 2013 at 02:58:01PM -0800, Dylan Baker wrote:
> forgot to mention this is available from my github:
> https://github.com/crymson/piglit.git simplify_subtests
>
>
> On Tuesday, November 05, 2013 10:34:49 AM Dylan Baker wrote:
> > This series attempts to fix the way piglit handles subtests. In
> > particular there are two problems with subtests:
> > 1. A subtest failure does not cause the test it is a part of to be a
> > failure
> > 2. Each subtest writes an additional copy of itself in the json
> >
> > Patch 1 attempts to correct point 1
> > The primary problem with tests not failing when their subtests fail is
> > that it is inconsistent with the rest of piglit; a group's status is
> > that of it's lowest test or subgroup.
> >
> > Patch 2 corrects point 2
> > Since tests with subtests are now groups, there is no need to write an
> > entry for each subtests. This reduces the size the json generated by
> > ~80% (That's 1/5 of the original size). This is significant not just for
> > the space savings, but because writing files is expensive.
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/piglit
-------------- next part --------------
{
"options": {
"profile": "tests/all_cl.tests",
"filter": [
"get-num-groups"
],
"exclude_filter": []
},
"name": "get-num-groups",
"lspci": "00:00.0 Host bridge: Advanced Micro Devices [AMD] Family 12h Processor Root Complex\n00:01.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI BeaverCreek [Radeon HD 6550D]\n00:01.1 Audio device: Advanced Micro Devices [AMD] nee ATI BeaverCreek HDMI Audio [Radeon HD 6500D and 6400G-6600G series]\n00:04.0 PCI bridge: Advanced Micro Devices [AMD] Family 12h Processor Root Port\n00:10.0 USB controller: Advanced Micro Devices [AMD] FCH USB XHCI Controller (rev 03)\n00:10.1 USB controller: Advanced Micro Devices [AMD] FCH USB XHCI Controller (rev 03)\n00:11.0 SATA controller: Advanced Micro Devices [AMD] FCH SATA Controller [IDE mode] (rev 40)\n00:12.0 USB controller: Advanced Micro Devices [AMD] FCH USB OHCI Controller (rev 11)\n00:12.2 USB controller: Advanced Micro Devices [AMD] FCH USB EHCI Controller (rev 11)\n00:13.0 USB controller: Advanced Micro Devices [AMD] FCH USB OHCI Controller (rev 11)\n00:13.2 USB controller: Advanced Micro Devices [AMD] FCH USB EHCI Controller (rev 11)\n00:14.0 SMBus: Advanced Micro Devices [AMD] FCH SMBus Controller (rev 13)\n00:14.1 IDE interface: Advanced Micro Devices [AMD] FCH IDE Controller\n00:14.2 Audio device: Advanced Micro Devices [AMD] FCH Azalia Controller (rev 01)\n00:14.3 ISA bridge: Advanced Micro Devices [AMD] FCH LPC Bridge (rev 11)\n00:14.4 PCI bridge: Advanced Micro Devices [AMD] FCH PCI Bridge (rev 40)\n00:14.5 USB controller: Advanced Micro Devices [AMD] FCH USB OHCI Controller (rev 11)\n00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 0 (rev 43)\n00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 1\n00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 2\n00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 3\n00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 4\n00:18.5 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 6\n00:18.6 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 5\n00:18.7 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 7\n01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 06)\n02:0e.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0)\n",
"glxinfo": "Error: unable to open display \n",
"tests": {
"Program/Execute/get-num-groups": {
"info": "Returncode: 1\n\nErrors:\n2 errors generated.\nCould not build program: CL_BUILD_PROGRAM_FAILURE\n\n\nOutput:\n## Test: get_num_groups (/home/tstellar/piglit/tests/cl/program/program-tester.c) ##\n\n# Platform supporting only version 1.1. Running test on that version.\n# Running on:\n# Platform: Default\n# Device: AMD SUMO\n# OpenCL version: 1.2\n# OpenCL C version: 1.1\n# Build options: -cl-std=CL1.1\nBuild log for device AMD SUMO:\n -------- \ninput.cl:134:9: error: use of type 'double' requires cl_khr_fp64 extension to be enabled\n double d = 5.0f + (double) out[0];\n ^\ninput.cl:134:29: error: use of type 'double' requires cl_khr_fp64 extension to be enabled\n double d = 5.0f + (double) out[0];\n ^\n\n -------- \n# Result:\n",
"dmesg": "",
"returncode": 1,
"note": "Returncode was 1",
"errors": [
"2 errors generated.",
"Could not build program: CL_BUILD_PROGRAM_FAILURE"
],
"command": "/home/tstellar/piglit/bin/cl-program-tester tests/cl/program/execute/get-num-groups.cl",
"result": "fail",
"time": 0.5378789901733398
}
},
"time_elapsed": 0.5464751720428467
}
-------------- next part --------------
{
"options": {
"profile": "tests/all_cl.tests",
"filter": [
"get-num-groups"
],
"exclude_filter": []
},
"name": "get-num-groups-good",
"lspci": "00:00.0 Host bridge: Advanced Micro Devices [AMD] Family 12h Processor Root Complex\n00:01.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI BeaverCreek [Radeon HD 6550D]\n00:01.1 Audio device: Advanced Micro Devices [AMD] nee ATI BeaverCreek HDMI Audio [Radeon HD 6500D and 6400G-6600G series]\n00:04.0 PCI bridge: Advanced Micro Devices [AMD] Family 12h Processor Root Port\n00:10.0 USB controller: Advanced Micro Devices [AMD] FCH USB XHCI Controller (rev 03)\n00:10.1 USB controller: Advanced Micro Devices [AMD] FCH USB XHCI Controller (rev 03)\n00:11.0 SATA controller: Advanced Micro Devices [AMD] FCH SATA Controller [IDE mode] (rev 40)\n00:12.0 USB controller: Advanced Micro Devices [AMD] FCH USB OHCI Controller (rev 11)\n00:12.2 USB controller: Advanced Micro Devices [AMD] FCH USB EHCI Controller (rev 11)\n00:13.0 USB controller: Advanced Micro Devices [AMD] FCH USB OHCI Controller (rev 11)\n00:13.2 USB controller: Advanced Micro Devices [AMD] FCH USB EHCI Controller (rev 11)\n00:14.0 SMBus: Advanced Micro Devices [AMD] FCH SMBus Controller (rev 13)\n00:14.1 IDE interface: Advanced Micro Devices [AMD] FCH IDE Controller\n00:14.2 Audio device: Advanced Micro Devices [AMD] FCH Azalia Controller (rev 01)\n00:14.3 ISA bridge: Advanced Micro Devices [AMD] FCH LPC Bridge (rev 11)\n00:14.4 PCI bridge: Advanced Micro Devices [AMD] FCH PCI Bridge (rev 40)\n00:14.5 USB controller: Advanced Micro Devices [AMD] FCH USB OHCI Controller (rev 11)\n00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 0 (rev 43)\n00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 1\n00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 2\n00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 3\n00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 4\n00:18.5 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 6\n00:18.6 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 5\n00:18.7 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 7\n01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 06)\n02:0e.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0)\n",
"glxinfo": "Error: unable to open display \n",
"tests": {
"Program/Execute/get-num-groups": {
"info": "Returncode: 0\n\nErrors:\n\n\nOutput:\n## Test: get_num_groups (/home/tstellar/piglit/tests/cl/program/program-tester.c) ##\n\n# Platform supporting only version 1.1. Running test on that version.\n# Running on:\n# Platform: Default\n# Device: AMD SUMO\n# OpenCL version: 1.2\n# OpenCL C version: 1.1\n# Build options: -cl-std=CL1.1\nProgram has been built successfully\n> Running kernel test: 1D, global_size 1 0 0, local_size 1 0 0\nUsing kernel fill1d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 1D, global_size 2 0 0, local_size 1 0 0\nUsing kernel fill1d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 1D, global_size 4 0 0, local_size 1 0 0\nUsing kernel fill1d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 1D, global_size 4 0 0, local_size 2 0 0\nUsing kernel fill1d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 1D, global_size 4 0 0, local_size 4 0 0\nUsing kernel fill1d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 2D, global_size 4 4 0, local_size 1 1 0\nUsing kernel fill2d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 2D, global_size 4 4 0, local_size 2 2 0\nUsing kernel fill2d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 2D, global_size 4 4 0, local_size 4 4 0\nUsing kernel fill2d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 3D, global_size 4 4 4, local_size 1 1 1\nUsing kernel fill3d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 3D, global_size 4 4 4, local_size 2 2 2\nUsing kernel fill3d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 3D, global_size 4 4 4, local_size 4 4 4\nUsing kernel fill3d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 3D, global_size 4 4 4, local_size 4 2 1\nUsing kernel fill3d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n>> All of the tests PASSED\n# Result:\n",
"dmesg": "",
"returncode": 0,
"subtest": {
"3D, global_size 4 4 4, local_size 4 2 1": "pass",
"3D, global_size 4 4 4, local_size 1 1 1": "pass",
"1D, global_size 4 0 0, local_size 2 0 0": "pass",
"2D, global_size 4 4 0, local_size 2 2 0": "pass",
"1D, global_size 4 0 0, local_size 4 0 0": "pass",
"1D, global_size 2 0 0, local_size 1 0 0": "pass",
"3D, global_size 4 4 4, local_size 4 4 4": "pass",
"1D, global_size 1 0 0, local_size 1 0 0": "pass",
"1D, global_size 4 0 0, local_size 1 0 0": "pass",
"2D, global_size 4 4 0, local_size 4 4 0": "pass",
"3D, global_size 4 4 4, local_size 2 2 2": "pass",
"2D, global_size 4 4 0, local_size 1 1 0": "pass"
},
"command": "/home/tstellar/piglit/bin/cl-program-tester tests/cl/program/execute/get-num-groups.cl",
"result": "pass",
"time": 0.6703798770904541
}
},
"time_elapsed": 0.679131031036377
}
-------------- next part --------------
{
"options": {
"profile": "tests/all_cl.tests",
"filter": [
"get-num-groups"
],
"exclude_filter": []
},
"name": "get-num-groups-2",
"lspci": "00:00.0 Host bridge: Advanced Micro Devices [AMD] Family 12h Processor Root Complex\n00:01.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI BeaverCreek [Radeon HD 6550D]\n00:01.1 Audio device: Advanced Micro Devices [AMD] nee ATI BeaverCreek HDMI Audio [Radeon HD 6500D and 6400G-6600G series]\n00:04.0 PCI bridge: Advanced Micro Devices [AMD] Family 12h Processor Root Port\n00:10.0 USB controller: Advanced Micro Devices [AMD] FCH USB XHCI Controller (rev 03)\n00:10.1 USB controller: Advanced Micro Devices [AMD] FCH USB XHCI Controller (rev 03)\n00:11.0 SATA controller: Advanced Micro Devices [AMD] FCH SATA Controller [IDE mode] (rev 40)\n00:12.0 USB controller: Advanced Micro Devices [AMD] FCH USB OHCI Controller (rev 11)\n00:12.2 USB controller: Advanced Micro Devices [AMD] FCH USB EHCI Controller (rev 11)\n00:13.0 USB controller: Advanced Micro Devices [AMD] FCH USB OHCI Controller (rev 11)\n00:13.2 USB controller: Advanced Micro Devices [AMD] FCH USB EHCI Controller (rev 11)\n00:14.0 SMBus: Advanced Micro Devices [AMD] FCH SMBus Controller (rev 13)\n00:14.1 IDE interface: Advanced Micro Devices [AMD] FCH IDE Controller\n00:14.2 Audio device: Advanced Micro Devices [AMD] FCH Azalia Controller (rev 01)\n00:14.3 ISA bridge: Advanced Micro Devices [AMD] FCH LPC Bridge (rev 11)\n00:14.4 PCI bridge: Advanced Micro Devices [AMD] FCH PCI Bridge (rev 40)\n00:14.5 USB controller: Advanced Micro Devices [AMD] FCH USB OHCI Controller (rev 11)\n00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 0 (rev 43)\n00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 1\n00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 2\n00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 3\n00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 4\n00:18.5 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 6\n00:18.6 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 5\n00:18.7 Host bridge: Advanced Micro Devices [AMD] Family 12h/14h Processor Function 7\n01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 06)\n02:0e.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0)\n",
"glxinfo": "Error: unable to open display \n",
"tests": {
"Program/Execute/get-num-groups": {
"info": "Returncode: 1\n\nErrors:\n\n\nOutput:\n## Test: get_num_groups (/home/tstellar/piglit/tests/cl/program/program-tester.c) ##\n\n# Platform supporting only version 1.1. Running test on that version.\n# Running on:\n# Platform: Default\n# Device: AMD SUMO\n# OpenCL version: 1.2\n# OpenCL C version: 1.1\n# Build options: -cl-std=CL1.1\nProgram has been built successfully\n> Running kernel test: 1D, global_size 1 0 0, local_size 1 0 0\nUsing kernel fill1d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 1D, global_size 2 0 0, local_size 1 0 0\nUsing kernel fill1d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 1D, global_size 4 0 0, local_size 1 0 0\nUsing kernel fill1d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 1D, global_size 4 0 0, local_size 2 0 0\nUsing kernel fill1d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 1D, global_size 4 0 0, local_size 4 0 0\nUsing kernel fill1d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 2D, global_size 4 4 0, local_size 1 1 0\nUsing kernel fill2d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\nExpecting 4 (0x4) with tolerance 0, but got 5 (0x5)\nError at int[1]\n Argument 0: FAIL\n> Running kernel test: 2D, global_size 4 4 0, local_size 2 2 0\nUsing kernel fill2d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\nExpecting 2 (0x2) with tolerance 0, but got 3 (0x3)\nError at int[1]\n Argument 0: FAIL\n> Running kernel test: 2D, global_size 4 4 0, local_size 4 4 0\nUsing kernel fill2d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\nExpecting 1 (0x1) with tolerance 0, but got 2 (0x2)\nError at int[1]\n Argument 0: FAIL\n> Running kernel test: 3D, global_size 4 4 4, local_size 1 1 1\nUsing kernel fill3d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 3D, global_size 4 4 4, local_size 2 2 2\nUsing kernel fill3d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 3D, global_size 4 4 4, local_size 4 4 4\nUsing kernel fill3d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n> Running kernel test: 3D, global_size 4 4 4, local_size 4 2 1\nUsing kernel fill3d\nSetting kernel arguments...\nRunning the kernel...\nValidating results...\n Argument 0: PASS\n>> Some or all of the tests FAILED\n# Result:\n",
"dmesg": "",
"returncode": 1,
"note": "Returncode was 1",
"subtest": {
"3D, global_size 4 4 4, local_size 4 2 1": "pass",
"3D, global_size 4 4 4, local_size 1 1 1": "pass",
"1D, global_size 4 0 0, local_size 2 0 0": "pass",
"2D, global_size 4 4 0, local_size 2 2 0": "fail",
"1D, global_size 4 0 0, local_size 4 0 0": "pass",
"1D, global_size 2 0 0, local_size 1 0 0": "pass",
"3D, global_size 4 4 4, local_size 4 4 4": "pass",
"1D, global_size 1 0 0, local_size 1 0 0": "pass",
"1D, global_size 4 0 0, local_size 1 0 0": "pass",
"2D, global_size 4 4 0, local_size 4 4 0": "fail",
"3D, global_size 4 4 4, local_size 2 2 2": "pass",
"2D, global_size 4 4 0, local_size 1 1 0": "fail"
},
"command": "/home/tstellar/piglit/bin/cl-program-tester tests/cl/program/execute/get-num-groups.cl",
"result": "fail",
"time": 0.6724631786346436
}
},
"time_elapsed": 0.6805989742279053
}
More information about the Piglit
mailing list