[Piglit] [PATCH] framework: Propagate crash to subtest.

[Dylan Baker]

Quoting Fabian Bieler (2018-01-20 05:38:45)
> If a subtest crashed it and all following subtests have the result notrun.
> Only the test as a whole has the result crash.
> This commit propagates the crash to the correct subtest.
> 
> Note that if the crash happened after the last subtest it is silently ignored.
> ---
> I like this approach. Personally, I would have added a NULL terminated array of char* to config that defaults to NULL and emitted the JSON structure in PIGLIT_GL_TEST_CONFIG_END, but that's just details.

I'm not a great C/C++ programmer, so if you have a better suggestion on how to
enumerate the subtests I'd be glad to do that instead.

> 
> Also this needs a little more work in the python framework: As is crashes don't show up in the result summary (regardless of whether the crash happens before or after the first call to piglit_report_subtest_result).
> 
> If we use an OrderedDict instead of a regular dict in patch [3/9] and as the container of Subtests (framework/results.py:44) we could deduce the crashing subtest (if the order is preserved between piglit_enumerate_subtests and test execution) with this patch.

You're right. I had a versions of this I was working on at one point, but
apparently lost when migrating machines or deleted :/

I think using an ordered dict would be better, in that case we can mark the
correct subtest as crashed during the test run and not need to mess with any
other code.

> 
> Note that this would differ from the previous behavior in that if a test crashes before a call to piglit_report_subtest_result (as is common in CL tests, I am told) the first subtest is marked as crashed and the whole test doesn't show up in the summary.
> 
> 
> I'll gladly help with the adaption of the remaining tests if needed.
> 
>  framework/results.py | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
> 
> diff --git a/framework/results.py b/framework/results.py
> index 99dd3735b..2e1b832fa 100644
> --- a/framework/results.py
> +++ b/framework/results.py
> @@ -69,6 +69,22 @@ class Subtests(collections.MutableMapping):
>          res['__type__'] = 'Subtests'
>          return res
>  
> +    def set_crash(self):
> +        """Find the first subtest marked as notrun and mark it as crashed.
> +
> +        If a subtest crashed it and all following subtests have the result
> +        notrun. Only the test as a whole has the result crash.
> +        This function propagates the crash to the correct subtest.
> +        Note that if the crash happened after the last subtest it is silently
> +        ignored.
> +        """
> +        if status.CRASH in six.itervalues(self):
> +            return
> +        for key, result in six.iteritems(self):
> +            if result == status.NOTRUN:
> +                self[key] = status.CRASH
> +                break
> +
>      @classmethod
>      def from_dict(cls, dict_):
>          if '__type__' in dict_:
> @@ -232,6 +248,8 @@ class TestResult(object):
>          # Set special instances
>          if 'subtests' in dict_:
>              inst.subtests = Subtests.from_dict(dict_['subtests'])
> +            if inst.result == status.CRASH:
> +                inst.subtests.set_crash()
>          if 'time' in dict_:
>              inst.time = TimeAttribute.from_dict(dict_['time'])
>  
> -- 
> 2.15.1
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: signature
URL: <https://lists.freedesktop.org/archives/piglit/attachments/20180122/865db38a/attachment.sig>