[Piglit] [PATCH] Add a test runner for the Vulkan CTS

[Dylan Baker]

Quoting Jason Ekstrand (2016-03-15 10:32:57)
> This is mostly a copy-and-paste of the original GLES3 dEQP runner with a
> few added tidbits that makes running the Vulkan CTS easier.  The two
> force-skip cases were very useful in the early days when lots of tests were
> still in development and may no longer be needed.  However, I have a
> feeling that we will yet want them.
> ---
>  tests/deqp_vk.py | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 91 insertions(+)
>  create mode 100644 tests/deqp_vk.py
> 
> diff --git a/tests/deqp_vk.py b/tests/deqp_vk.py
> new file mode 100644
> index 0000000..bc84229
> --- /dev/null
> +++ b/tests/deqp_vk.py
> @@ -0,0 +1,91 @@
> +# Copyright 2014, 2015 Intel Corporation
> +#
> +# Permission is hereby granted, free of charge, to any person obtaining a copy
> +# of this software and associated documentation files (the "Software"), to deal
> +# in the Software without restriction, including without limitation the rights
> +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
> +# copies of the Software, and to permit persons to whom the Software is
> +# furnished to do so, subject to the following conditions:
> +#
> +# The above copyright notice and this permission notice shall be included in
> +# all copies or substantial portions of the Software.
> +#
> +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
> +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
> +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> +# SOFTWARE.
> +
> +"""Piglit integrations for dEQP VK tests."""
> +
> +import xml.etree.cElementTree as ET
> +import re
> +
> +from framework.test import deqp
> +
> +__all__ = ['profile']
> +
> +# Path to the deqp-gles3 executable.
> +_DEQP_VK_EXE = deqp.get_option('PIGLIT_DEQP_VK_EXE',
> +                               ('deqp-vk', 'exe'))
> +
> +_EXTRA_ARGS = deqp.get_option('PIGLIT_DEQP_VK_EXTRA_ARGS',
> +                              ('deqp-vk', 'extra_args'),
> +                              default='').split()
> +
> +
> +def _get_test_case(root, root_group, outputfile):
> +    """Parser the test case list of Google Android CTS,
> +    and store the test case list to dEQP-VK-cases.txt
> +    """
> +    for child in root:
> +        root_group.append(child.get('name'))
> +        if child.tag == "Test":
> +            outputfile.write('TEST: {}\n'.format('.'.join(root_group)))
> +            del root_group[-1]
> +        else:
> +            _get_test_case(child, root_group, outputfile)
> +            del root_group[-1]
> +
> +
> +def _load_test_hierarchy(mustpass, case_list):
> +    """Google have added a subset of dEQP to CTS test, the case list is stored
> +    at some xml files. Such as: com.drawelements.deqp.gles3.xml This function
> +    is used to parser the file, and generate a new dEQP-VK-cases.txt which
> +    only contain the subset of dEQP.
> +    """
> +    tree = ET.parse(mustpass)
> +    root = tree.getroot()
> +    root_group = []
> +    with open(case_list, 'w') as f:
> +        _get_test_case(root, root_group, f)
> +
> +_deqp_assert = re.compile(r'deqp-vk: external/vulkancts/.*: Assertion `.*\' failed.')
> +
> +class DEQPVKTest(deqp.DEQPBaseTest):
> +    timeout = 60
> +    deqp_bin = _DEQP_VK_EXE
> +    extra_args = [x for x in _EXTRA_ARGS if not x.startswith('--deqp-case')]
> +
> +
> +    def __init__(self, *args, **kwargs):
> +        super(DEQPVKTest, self).__init__(*args, **kwargs)
> +
> +    def interpret_result(self):
> +        if 'Failed to compile shader at vkGlslToSpirV' in self.result.out:
> +            self.result.result = 'skip'
> +            self.result.out += '\n\nMarked as skip because GLSLang failed to compile shaders'
> +        elif _deqp_assert.search(self.result.err):
> +            self.result.result = 'skip'
> +            self.result.out += '\n\nMarked as skip because of a internal dEQP assertion'
> +        else:
> +            super(DEQPVKTest, self).interpret_result()
> +
> +
> +profile = deqp.make_profile(  # pylint: disable=invalid-name
> +    deqp.iter_deqp_test_cases(
> +        deqp.gen_caselist_txt(_DEQP_VK_EXE, 'dEQP-VK-cases.txt',
> +                              _EXTRA_ARGS)),
> +    DEQPVKTest)
> -- 
> 2.5.0.400.gff86faf
> 
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/piglit

This looks fine, there are a few little tid-bits I'll update for
upstream piglit changes and push.

Dylan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: signature
URL: <https://lists.freedesktop.org/archives/piglit/attachments/20160315/d5267ba4/attachment.sig>