[Piglit] [PATCH] deqp: Add test profile for external dEQP-GLES3 tests (v3)
Mason, Michael W
michael.w.mason at intel.com
Sun Dec 14 08:29:32 PST 2014
> -----Original Message-----
> From: Dylan Baker [mailto:baker.dylan.c at gmail.com]
> Sent: Friday, December 12, 2014 6:04 PM
> To: piglit at lists.freedesktop.org
> Cc: Mason, Michael W; Chad Versace
> Subject: Re: [Piglit] [PATCH] deqp: Add test profile for external dEQP-GLES3 tests (v3)
>
> On Saturday, December 13, 2014 12:53:42 AM Mason, Michael W wrote:
> > A couple questions...
> >
> > First, deqp-gles3 has many command line options. Is there any way in piglit to pass them to deqp-gles3 from the piglit
> command line? I don't see a way. One option in particular is important when testing on Freon. I have to pass "--deqp-gl-
> config-name=rgba8888d24s8" to get the images to display full screen. No doubt we'll want to use other options in the
> future.
>
> There is no way currently. It wouldn't be that hard to implement the feature however. The other option is to do what we
> do for several of the integrated suites and use a mixture of environment variables and piglit.conf settings, depending on
> how often you want to change those settings.
Using the env variable/piglit.conf method to pass options to deqp-gles3 works great.
See my suggestions below.
>
> >
> > Second, how do you kill a piglit test run once it's started? I tried multiple ctrl-C's without success. I couldn't get it to stop.
> Is this a piglit limitation or something to do with the piglit/deqp combo?
> >
> > BTW, the latest patch does work, but I'd like to understand if the above limitations are expected or not before giving it my
> blessing.
>
> As Matt pointed out ctrl+\ will kill all tests. ctrl+c will kill a single running test.
> >
> > Mike
> >
> > -----Original Message-----
> > From: Chad Versace [mailto:chad.versace at linux.intel.com]
> > Sent: Wednesday, December 10, 2014 9:39 AM
> > To: piglit at lists.freedesktop.org
> > Cc: Chad Versace; Baker, DylanX C; Mason, Michael W
> > Subject: [PATCH] deqp: Add test profile for external dEQP-GLES3 tests
> > (v3)
> >
> > Google opensourced the drawElements Quality Product Testsuite (dEQP) as part of the Android Lollipop release. It's git
> repo lives in the Android tree at [https://android.googlesource.com/platform/external/deqp/].
> >
> > This patch adds a new module, deqp_gles3.py, that runs the dEQP-GLES3 tests. It queries the 'deqp-gles3' executable at
> runtime for the list of testcases. The module attempts queries the 'deqp-gles3' executable only if the environment
> variable PIGLIT_DEQP_GLES3_EXE or the piglit.conf option deqp-gles3.exe is set.
> >
> > Why do we need to use Pigit as the testrunner for dEQP? (After all,
> > dEQP has its own testrunner). Because dEQP runs all tests in a single
> > process. If test
> > 17530 of 55409 crashes, then the dEQP testrunner crashes and the remaining tests never run. Piglit doesn't suffer from
> that problem, because it runs each test as a separate process.
> >
> > Tested on Intel Ivybridge by running the command
> > piglit run -t dEQP-GLES3/info/vendor \
> > tests/deqp_gles3.py tests/quick.py results with and without PIGLIT_DEQP_GLES3_EXE set and with and without
> piglit.conf:deqp-gles.exe set.
> >
> > Cc: Dylan Baker <dylanx.c.baker at intel.com>
> > Cc: Michael W Mason <michael.w.mason at intel.com>
> > Signed-off-by: Chad Versace <chad.versace at linux.intel.com>
> > ---
> >
> > v2:
> > - Remove unused imports (dcbaker)
> > - Remove extra newlines (dcbaker)
> > - Don't use 'global' keyword (dcbaker)
> > - Use global constant for DEQP_GLES3_EXE (dcbaker)
> > - Fix DEQPTest.interpret_result to correctly parse test output. (mwmason)
> > - Use 'super' keyword (dcbaker)
> >
> > v3:
> > - Fix misnamed variable s/caselist_txt/caselist_path/. (mwmason)
> >
> > piglit.conf.example | 7 +++
> > tests/deqp_gles3.py | 136
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++
> > 2 files changed, 143 insertions(+)
> > create mode 100644 tests/deqp_gles3.py
> >
> > diff --git a/piglit.conf.example b/piglit.conf.example index
> > a864c01..0ac0f4f 100644
> > --- a/piglit.conf.example
> > +++ b/piglit.conf.example
> > @@ -33,6 +33,13 @@ bindir=/home/usr/oclconform testA testB
> >
> > +;[deqp-gles3]
> > +;
> > +; Path to the deqp-gles3 executable. You can also set this with the
> > +environment ; variable PIGLIT_DEQP_GLES3_EXE. Piglit will run the
> > +dEQP-GLES3 tests if and ; only if this option is set.
> > +;exe=/home/knuth/deqp/modules/gles3/deqp-gles3
> > +
> > ; Section for specific oclconform test. One of these sections is required for ; each test list in the oclconform section and
> must be called:
> > ; oclconform-$testname
> > diff --git a/tests/deqp_gles3.py b/tests/deqp_gles3.py new file mode
> > 100644 index 0000000..24035c6
> > --- /dev/null
> > +++ b/tests/deqp_gles3.py
> > @@ -0,0 +1,136 @@
> > +# Copyright 2014 Intel Corporation
> > +#
> > +# Permission is hereby granted, free of charge, to any person
> > +obtaining a copy # of this software and associated documentation
> > +files (the "Software"), to deal # in the Software without
> > +restriction, including without limitation the rights # to use, copy,
> > +modify, merge, publish, distribute, sublicense, and/or sell # copies
> > +of the Software, and to permit persons to whom the Software is # furnished to do so, subject to the following
> conditions:
> > +#
> > +# The above copyright notice and this permission notice shall be
> > +included in # all copies or substantial portions of the Software.
> > +#
> > +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> > +EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> > +MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND
> > +NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS
> > +BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN
> > +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN
> > +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE # SOFTWARE.
> > +
> > +import ConfigParser
> > +import os
> > +import subprocess
> > +
> > +# Piglit modules
> > +import framework
> > +from framework.profile import Test, TestProfile
> > +
> > +__all__ = ['profile']
> > +
> > +def get_option(env_varname, config_option):
> > + """Query the given environment variable and then piglit.conf for the
> > + option. Return None if the option is unset.
> > + """
> > + opt = os.environ.get(env_varname, None)
> > + if opt is not None:
> > + return opt
> > +
> > + try:
> > + opt = framework.core.PIGLIT_CONFIG.get(config_option[0],
> > + config_option[1])
> > + except ConfigParser.NoSectionError, ConfigParser.NoOptionError:
> > + pass
> > +
> > + return opt
> > +
> > +
> > +# Path to the deqp-gles3 executable.
> > +DEQP_GLES3_EXE = get_option('PIGLIT_DEQP_GLES3_EXE', ('deqp-gles3', 'exe'))
Add an environment variable for options to pass to deqp-gles3:
# Command line options to pass to deqp-gles3 executable.
DEQP_GLES3_OPT = get_option('PIGLIT_DEQP_GLES3_OPT', ('deqp-gles3', 'options'))
> > +
> > +
> > +def gen_caselist_txt():
> > + """Generate dEQP-GLES3-cases.txt and return its path."""
> > + # dEQP is stupid (2014-12-07):
> > + # 1. To generate the caselist file, dEQP requires that the process's
> > + # current directory must be that same as that of the executable.
> > + # Otherwise, it fails to find its data files.
> > + # 2. dEQP creates the caselist file in the process's current directory
> > + # and provides no option to change its location.
> > + # 3. dEQP creates a GL context when generating the caselist. Therefore,
> > + # the caselist must be generated on the test target rather than the
> > + # build host. In other words, when the build host and test target
> > + # differ then we cannot pre-generate the caselist on the build host:
> > + # we must *dynamically* generate it during the testrun.
> > + basedir = os.path.dirname(DEQP_GLES3_EXE)
> > + caselist_path = os.path.join(basedir, 'dEQP-GLES3-cases.txt')
> > + subprocess.check_call(
> > + [DEQP_GLES3_EXE, '--deqp-runmode=txt-caselist'],
> > + cwd=basedir)
> > + assert os.path.exists(caselist_path)
> > + return caselist_path
> > +
> > +
> > +def iter_deqp_test_cases():
> > + """Iterate over original dEQP GLES3 testcase names."""
> > + caselist_path = gen_caselist_txt()
> > + with open(caselist_path) as caselist_file:
> > + for i, line in enumerate(caselist_file):
> > + if line.startswith('GROUP:'):
> > + continue
> > + elif line.startswith('TEST:'):
> > + yield line[len('TEST:'):].strip()
> > + else:
> > + raise Exception('{0}:{1}: ill-formed
> > +line'.format(caselist_path, i))
> > +
> > +
> > +class DEQPTest(Test):
> > +
> > + RESULT_MAP = {"Pass" : "pass",
> > + "Fail" : "fail",
> > + "QualityWarning" : "warn",
> > + "InternalError" : "fail",
> > + "Crash" : "crash",
> > + "NotSupported" : "skip"}
> > +
> > + def __init__(self, case_name):
> > + command = [
> > + DEQP_GLES3_EXE,
> > + '--deqp-case=' + case_name,
> > + '--deqp-visibility=hidden',
> > + ]
Change command to this:
command = [
DEQP_GLES3_EXE,
'--deqp-case=' + case_name,
DEQP_GLES3_OPT
]
The --deqp-visibility option isn't required, so let's leave that to the user.
> > + super(DEQPTest, self).__init__(command)
> > +
> > + # dEQP's working directory must be the same as that of the executable,
> > + # otherwise it cannot find its data files (2014-12-07).
> > + self.cwd = os.path.dirname(DEQP_GLES3_EXE)
> > +
> > + def interpret_result(self):
> > + if self.result['returncode'] != 0:
> > + self.result['result'] = 'fail'
> > + return
> > +
> > + for line in self.result['out'].split('\n'):
> > + line = line.lstrip()
> > + for (k, v) in DEQPTest.RESULT_MAP.iteritems():
> > + if line.startswith(k):
> > + self.result['result'] = v
> > + return
> > +
> > + # We failed to parse the test output. Fallback to 'fail'.
> > + self.result['result'] = 'fail'
> > +
> > +def add_tests():
> > + if DEQP_GLES3_EXE is None:
> > + return
> > +
> > + for deqp_testname in iter_deqp_test_cases():
> > + # dEQP uses '.' as the testgroup separator.
> > + piglit_testname = deqp_testname.replace('.', '/')
> > +
> > + test = DEQPTest(deqp_testname)
> > + profile.test_list[piglit_testname] = test
> > +
> > +
> > +profile = TestProfile()
> > +add_tests()
> > --
> > 2.2.0
> >
> > _______________________________________________
> > Piglit mailing list
> > Piglit at lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/piglit
> >
More information about the Piglit
mailing list