[Piglit] [PATCH v4] framework: Add a vulkan tests profile

Wed Nov 7 21:31:53 UTC 2018

Quoting Neil Roberts (2018-11-05 02:04:44)
> This searches for files named *.vk_shader_test in the tests/vulkan
> directory and runs them with VkRunner. VkRunner is executed as an
> external dependency. It is found either with the vkrunner:bin config
> option, by setting the PIGLIT_VKRUNNER_BINARY environment variable, or
> just in the search path.
> 
> v2: Move VkShaderTest to piglit_test.py and rename to VkRunnerTest.
>     Add future imports. Remove unused import.
> v3: Support the PIGLIT_VKRUNNER_BINARY variable to specify the
>     location of VkRunner.
> v4: Add documentation to the README. Add an option in piglit.conf to
>     set the binary location. (Suggested by Samuel Iglesias)
> ---
>  README.md                     | 17 ++++++++++++++++-
>  framework/test/piglit_test.py | 23 +++++++++++++++++++++++
>  piglit.conf.example           |  4 ++++
>  tests/vulkan.py               | 33 +++++++++++++++++++++++++++++++++
>  4 files changed, 76 insertions(+), 1 deletion(-)
>  create mode 100644 tests/vulkan.py
> 
> diff --git a/README.md b/README.md
> index 2ae0febc6..c75c5f6c3 100644
> --- a/README.md
> +++ b/README.md
> @@ -47,6 +47,8 @@ Optionally, you can install the following:
>      (https://simplejson.readthedocs.org/en/latest/)
>    - jsonstreams. A JSON stream writer for python.
>      (https://jsonstreams.readthedocs.io/en/stable/)
> +  - VkRunner. A shader script testing tool for Vulkan.
> +    (https://github.com/igalia/vkrunner)
>  
>  For Python 2.x you can install the following to add features, these are
>  unnecessary for python3:
> @@ -285,6 +287,13 @@ behaves.
>      When this variable is true in python then any timeouts given by tests
>      will be ignored, and they will run until completion or they are killed.
>  
> +  - `PIGLIT_VKRUNNER_BINARY`
> +
> +    Can be used to override the path to the vkrunner executable for
> +    running Vulkan shader tests. Alternatively the config option
> +    vkrunner:bin can be used instead. If neither are set then vkrunner
> +    will be searched for in the search path.
> +
>  
>  ### 3.2 Note
>  
> @@ -338,7 +347,13 @@ The following test sets are currently available:
>      opencv and oclconform.
>  
>  
> -### 4.3 External Integration
> +### 4.3 Vulkan tests
> +
> +  - **vulkan.py** This suite contains all Vulkan tests. Note that
> +    currently all of the Vulkan tests require VkRunner. If it is not
> +    installed then all of the tests will be skipped.
> +
> +### 4.4 External Integration
>  
>    - **xts.py** Support for running the X Test Suite using piglit.
>    - **igt.py** Support for running Intel-gpu-tools test suite using piglit.
> diff --git a/framework/test/piglit_test.py b/framework/test/piglit_test.py
> index f52915d18..c80e355d1 100644
> --- a/framework/test/piglit_test.py
> +++ b/framework/test/piglit_test.py
> @@ -44,6 +44,7 @@ __all__ = [
>      'PiglitCLTest',
>      'PiglitGLTest',
>      'PiglitBaseTest',
> +    'VkRunnerTest',
>      'CL_CONCURRENT',
>      'ROOT_DIR',
>      'TEST_BIN_DIR',
> @@ -229,3 +230,25 @@ class CLProgramTester(PiglitCLTest):
>          command = super(CLProgramTester, self).command
>          command.insert(1, os.path.join(ROOT_DIR, self.filename))
>          return command
> +
> +
> +class VkRunnerTest(PiglitBaseTest):
> +    """ Make a PiglitTest instance for a VkRunner shader test file """
> +
> +    def __init__(self, filename):
> +        vkrunner_bin = os.environ.get('PIGLIT_VKRUNNER_BINARY')
> +
> +        if vkrunner_bin is None:
> +            vkrunner_bin = core.PIGLIT_CONFIG.safe_get(
> +                'vkrunner', 'bin', fallback='vkrunner')
> +
> +        super(VkRunnerTest, self).__init__(
> +            [vkrunner_bin, filename],
> +            run_concurrent=True)
> +
> +    @PiglitBaseTest.command.getter
> +    def command(self):
> +        # This is overriden because we don’t want PiglitBaseTest to
> +        # prepend TEST_BIN_DIR so that it will look for vkrunner in
> +        # the search path.
> +        return self._command
> diff --git a/piglit.conf.example b/piglit.conf.example
> index 5d1ff5474..1877187df 100644
> --- a/piglit.conf.example
> +++ b/piglit.conf.example
> @@ -187,6 +187,10 @@ run_test=./%(test_name)s
>  ; Default: True
>  ;process isolation=True
>  
> +[vkrunner]
> +; Path to the VkRunner executable
> +; bin=/home/neil/local/bin/vkrunner
> +
>  [expected-failures]
>  ; Provide a list of test names that are expected to fail.  These tests
>  ; will be listed as passing in JUnit output when they fail.  Any
> diff --git a/tests/vulkan.py b/tests/vulkan.py
> new file mode 100644
> index 000000000..7058f3108
> --- /dev/null
> +++ b/tests/vulkan.py
> @@ -0,0 +1,33 @@
> +#!/usr/bin/env python
> +# -*- coding: utf-8 -*-
> +"""All Vulkan tests that come with piglit, using default settings."""
> +
> +from __future__ import (
> +    absolute_import, division, print_function, unicode_literals
> +)
> +
> +import os
> +
> +from framework.profile import TestProfile
> +from framework import grouptools
> +from framework.test.piglit_test import VkRunnerTest
> +from .py_modules.constants import TESTS_DIR, GENERATED_TESTS_DIR
> +
> +__all__ = ['profile']
> +
> +profile = TestProfile()
> +
> +# Find and add all shader tests.
> +for basedir in [TESTS_DIR, GENERATED_TESTS_DIR]:
> +    _basedir = os.path.join(basedir, 'vulkan')
> +    for dirpath, _, filenames in os.walk(_basedir):
> +        groupname = grouptools.from_path(os.path.relpath(dirpath, _basedir))
> +        for filename in filenames:
> +            testname, ext = os.path.splitext(filename)
> +            if ext != '.vk_shader_test':
> +                continue
> +            test = VkRunnerTest(os.path.join(dirpath, filename))
> +            group = grouptools.join(groupname, testname)
> +            assert group not in profile.test_list, group
> +
> +            profile.test_list[group] = test
> -- 

I'm find with keeping vkrunner separate from piglit, we can always merge it in
tree later if we decide that would be better.

The one thing I'm not seeing is xml generation, there should be something in
tests/CMakeLists.no_api.txt for generating an XML file. It's not strictly
required, but it makes the runner start time much better. I think all you need
is to add a static-vk-tests target (cut-n-paste + some small edits to
find_static_tests.py), and a call to piglit_generate_xml.

I'm okay with that as a follow up change if you'd like to land this now.

Reviewed-by: Dylan Baker <dylan at pnwbakers.com>

Dylan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: signature
URL: <https://lists.freedesktop.org/archives/piglit/attachments/20181107/8c785119/attachment-0001.sig>