[Piglit] [PATCH] deqp: Add test profile for external dEQP-GLES3 tests

[Chad Versace]

On 12/10/2014 09:03 AM, Mike Mason wrote:
> Hi Chad,
> 
> See comments below...

Mike, I sent v2 before I saw your reply. I'll respin a v3 shortly that takes
your comments into account.
 

>> +def iter_deqp_test_cases():
>> +    """Iterate over original dEQP GLES3 testcase names."""
>> +    caselist_path = gen_caselist_txt()
>> +    with open(caselist_path) as caselist_file:
>> +        for i, line in enumerate(caselist_file):
>> +            if line.startswith('GROUP:'):
>> +                continue
>> +            elif line.startswith('TEST:'):
>> +                yield line[len('TEST:'):].strip()
>> +            else:
>> +                raise Exception('{0}:{1}: ill-formed line'.format(caselist_txt, i))
>                                                                      ^^^^^^^^^^^^
> caselist_txt doesn't exist.  Did you mean to use line or caselist_path?

Thanks. Will fix in v3.


>> +class DEQPTest(framework.profile.Test):
>> +
>> +    def __init__(self, case_name):
>> +        command = [
>> +            get_deqp_gles3_exe(),
>> +            '--deqp-case=' + case_name,
>> +            '--deqp-visibility=hidden',
>> +        ]
>> +        framework.profile.Test.__init__(self, command)
>> +
>> +        # dEQP's working directory must be the same as that of the executable,
>> +        # otherwise it cannot find its data files (2014-12-07).
>> +        self.cwd = os.path.dirname(get_deqp_gles3_exe())
>> +
>> +    def interpret_result(self):
>> +        if self.result['returncode'] == 0:
>> +            self.result['result'] = 'pass'
>> +        else:
>> +            self.result['result'] = 'fail'
> 
> The script interprets deqp results is incorrectly. If the binary
> runs and exits cleanly it returns 0 regardless of the actual test result.
> I verified this by testing the return code of a failed test.
> The real result is in self.result[‘out’].
> 
> Here’s a version of interpret_result that takes that into account:
> 
> def interpret_result(self):
>     result_map = {"Pass"           : "pass",
>                   "Fail"           : "fail",
>                   "QualityWarning" : "fail",
>                   "InternalError"  : "fail",
>                   "Crash"          : "fail",
>                   "NotSupported"   : "skip"}
> 
>     if self.result['returncode'] != 0:
>         self.result['result'] = 'fail'
>         return
> 
>     for line in self.result['out'].split('\n'):
>         line = line.lstrip()
>         for result in result_map:
>             if line.startswith(result):
>                 self.result['result'] = result_map[result]
>                 return

In v2, I made some slight changes to your suggested code:
  - Since Test.interpret_result is called so frequently, I moved
    the contruction of the result_map dictionary from per-call
    to once-on-module-load by redeclaring it as a class variable.
  - I parse "Crash" -> "crash" and "QualityWarning" -> "warn".
    Piglit recognizes those result types and colors them black and
    yellow (?) in its HTML output.
  - If DEQPTest.interpret_result fails to parse the test output,
    I fallback to setting self.result['result'] = fail.

Thanks for catching my mistake. Please review the changes I made
for correctness.