[Piglit] [PATCH] deqp: Add test profile for external dEQP-GLES3 tests (v4)
Mason, Michael W
michael.w.mason at intel.com
Wed Dec 17 12:58:00 PST 2014
Reviewed-by: Michael W Mason <michael.w.mason at intel.com>
I ran this on link-freon and it worked as expected. Thanks Chad!
> -----Original Message-----
> From: Chad Versace [mailto:chad.versace at linux.intel.com]
> Sent: Tuesday, December 16, 2014 1:49 PM
> To: piglit at lists.freedesktop.org
> Cc: Chad Versace; Baker, DylanX C; Mason, Michael W
> Subject: [PATCH] deqp: Add test profile for external dEQP-GLES3 tests (v4)
>
> Google opensourced the drawElements Quality Product Testsuite (dEQP) as part of the Android Lollipop release. It's git
> repo lives in the Android tree at [https://android.googlesource.com/platform/external/deqp/].
>
> This patch adds a new module, deqp_gles3.py, that runs the dEQP-GLES3 tests. It queries the 'deqp-gles3' executable at
> runtime for the list of testcases. The module attempts queries the 'deqp-gles3' executable only if the environment
> variable PIGLIT_DEQP_GLES3_EXE or the piglit.conf option deqp-gles3.exe is set.
>
> Why do we need to use Pigit as the testrunner for dEQP? (After all, dEQP has its own testrunner). Because dEQP runs all
> tests in a single process. If test
> 17530 of 55409 crashes, then the dEQP testrunner crashes and the remaining tests never run. Piglit doesn't suffer from that
> problem, because it runs each test as a separate process.
>
> Tested on Intel Ivybridge by running the command
> piglit run -t dEQP-GLES3/info/vendor \
> tests/deqp_gles3.py tests/quick.py results with and without PIGLIT_DEQP_GLES3_EXE set and with and without
> piglit.conf:deqp-gles.exe set. Also tested with various combinations of valid and invalid values for
> PIGLIT_DEQP_GLES_EXTRA_ARGS and piglit.conf:deqp-gles.extra_args.
>
> Cc: Dylan Baker <dylanx.c.baker at intel.com>
> Cc: Michael W Mason <michael.w.mason at intel.com>
> Signed-off-by: Chad Versace <chad.versace at linux.intel.com>
> ---
>
> v2:
> - Remove unused imports (dcbaker)
> - Remove extra newlines (dcbaker)
> - Don't use 'global' keyword (dcbaker)
> - Use global constant for DEQP_GLES3_EXE (dcbaker)
> - Fix DEQPTest.interpret_result to correctly parse test output. (mwmason)
> - Use 'super' keyword (dcbaker)
>
> v3:
> - Fix misnamed variable s/caselist_txt/caselist_path/. (mwmason)
>
> v4:
> - Add option piglit.conf:extra_args and env var
> PIGLIT_DEQP_GLES3_EXTRA_ARGS. (requested by mwmason)
> - Remove '--deqp-visibility=hidden'. (mwmason)
> - Fixed exception handling in func get_option.
>
>
> piglit.conf.example | 12 +++++
> tests/deqp_gles3.py | 139 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 151 insertions(+)
> create mode 100644 tests/deqp_gles3.py
>
> diff --git a/piglit.conf.example b/piglit.conf.example index a864c01..b3869c2 100644
> --- a/piglit.conf.example
> +++ b/piglit.conf.example
> @@ -33,6 +33,18 @@ bindir=/home/usr/oclconform testA testB
>
> +;[deqp-gles3]
> +;
> +; Path to the deqp-gles3 executable. You can also set this with the
> +environment ; variable PIGLIT_DEQP_GLES3_EXE. Piglit will run the
> +dEQP-GLES3 tests if and ; only if this option is set.
> +;exe=/home/knuth/deqp/modules/gles3/deqp-gles3
> +;
> +; Space-separated list of extra command line arguments for deqp-gles3.
> +The ; option is not required. The environment variable
> +PIGLIT_DEQP_GLES3_EXTRA_ARGS ; overrides the value set here.
> +;extra_args=--deqp-visibility hidden
> +
> ; Section for specific oclconform test. One of these sections is required for ; each test list in the oclconform section and
> must be called:
> ; oclconform-$testname
> diff --git a/tests/deqp_gles3.py b/tests/deqp_gles3.py new file mode 100644 index 0000000..b1bb33b
> --- /dev/null
> +++ b/tests/deqp_gles3.py
> @@ -0,0 +1,139 @@
> +# Copyright 2014 Intel Corporation
> +#
> +# Permission is hereby granted, free of charge, to any person obtaining
> +a copy # of this software and associated documentation files (the
> +"Software"), to deal # in the Software without restriction, including
> +without limitation the rights # to use, copy, modify, merge, publish,
> +distribute, sublicense, and/or sell # copies of the Software, and to
> +permit persons to whom the Software is # furnished to do so, subject to the following conditions:
> +#
> +# The above copyright notice and this permission notice shall be
> +included in # all copies or substantial portions of the Software.
> +#
> +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> +EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> +MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND
> +NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS
> +BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN
> +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN
> +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE # SOFTWARE.
> +
> +import ConfigParser
> +import os
> +import subprocess
> +
> +# Piglit modules
> +import framework
> +from framework.profile import Test, TestProfile
> +
> +__all__ = ['profile']
> +
> +def get_option(env_varname, config_option):
> + """Query the given environment variable and then piglit.conf for the
> + option. Return None if the option is unset.
> + """
> + opt = os.environ.get(env_varname, None)
> + if opt is not None:
> + return opt
> +
> + try:
> + opt = framework.core.PIGLIT_CONFIG.get(config_option[0],
> + config_option[1])
> + except ConfigParser.NoSectionError:
> + pass
> + except ConfigParser.NoOptionError:
> + pass
> +
> + return opt
> +
> +
> +# Path to the deqp-gles3 executable.
> +DEQP_GLES3_EXE = get_option('PIGLIT_DEQP_GLES3_EXE', ('deqp-gles3',
> +'exe'))
> +
> +DEQP_GLES3_EXTRA_ARGS = get_option('PIGLIT_DEQP_GLES3_EXTRA_ARGS',
> + ('deqp-gles3', 'extra_args'))
> +
> +
> +def gen_caselist_txt():
> + """Generate dEQP-GLES3-cases.txt and return its path."""
> + # dEQP is stupid (2014-12-07):
> + # 1. To generate the caselist file, dEQP requires that the process's
> + # current directory must be that same as that of the executable.
> + # Otherwise, it fails to find its data files.
> + # 2. dEQP creates the caselist file in the process's current directory
> + # and provides no option to change its location.
> + # 3. dEQP creates a GL context when generating the caselist. Therefore,
> + # the caselist must be generated on the test target rather than the
> + # build host. In other words, when the build host and test target
> + # differ then we cannot pre-generate the caselist on the build host:
> + # we must *dynamically* generate it during the testrun.
> + basedir = os.path.dirname(DEQP_GLES3_EXE)
> + caselist_path = os.path.join(basedir, 'dEQP-GLES3-cases.txt')
> + subprocess.check_call(
> + [DEQP_GLES3_EXE, '--deqp-runmode=txt-caselist'],
> + cwd=basedir)
> + assert os.path.exists(caselist_path)
> + return caselist_path
> +
> +
> +def iter_deqp_test_cases():
> + """Iterate over original dEQP GLES3 testcase names."""
> + caselist_path = gen_caselist_txt()
> + with open(caselist_path) as caselist_file:
> + for i, line in enumerate(caselist_file):
> + if line.startswith('GROUP:'):
> + continue
> + elif line.startswith('TEST:'):
> + yield line[len('TEST:'):].strip()
> + else:
> + raise Exception('{0}:{1}: ill-formed
> +line'.format(caselist_path, i))
> +
> +
> +class DEQPTest(Test):
> +
> + __RESULT_MAP = {"Pass" : "pass",
> + "Fail" : "fail",
> + "QualityWarning" : "warn",
> + "InternalError" : "fail",
> + "Crash" : "crash",
> + "NotSupported" : "skip"}
> +
> + def __init__(self, case_name):
> + command = [DEQP_GLES3_EXE, '--deqp-case=' + case_name]
> + if DEQP_GLES3_EXTRA_ARGS is not None:
> + command.extend(DEQP_GLES3_EXTRA_ARGS.split())
> + super(DEQPTest, self).__init__(command)
> +
> + # dEQP's working directory must be the same as that of the executable,
> + # otherwise it cannot find its data files (2014-12-07).
> + self.cwd = os.path.dirname(DEQP_GLES3_EXE)
> +
> + def interpret_result(self):
> + if self.result['returncode'] != 0:
> + self.result['result'] = 'fail'
> + return
> +
> + for line in self.result['out'].split('\n'):
> + line = line.lstrip()
> + for k, v in DEQPTest.__RESULT_MAP.iteritems():
> + if line.startswith(k):
> + self.result['result'] = v
> + return
> +
> + # We failed to parse the test output. Fallback to 'fail'.
> + self.result['result'] = 'fail'
> +
> +def add_tests():
> + if DEQP_GLES3_EXE is None:
> + return
> +
> + for deqp_testname in iter_deqp_test_cases():
> + # dEQP uses '.' as the testgroup separator.
> + piglit_testname = deqp_testname.replace('.', '/')
> +
> + test = DEQPTest(deqp_testname)
> + profile.test_list[piglit_testname] = test
> +
> +
> +profile = TestProfile()
> +add_tests()
> --
> 2.2.0
More information about the Piglit
mailing list