[Piglit] [PATCH] framework/backends/junit: Report expected failures/crashes as skipped.

[Dylan Baker]

Acked-by: Dylan Baker <baker.dylan.c at gmail.com>

On Tue, Mar 03, 2015 at 02:47:59PM +0000, Jose Fonseca wrote:
> I recently tried the junit backend's ability to ignore expected
> failures/crashes and found it a godsend -- instead of having to look as
> test graph results periodically, I can just tell jenkins to email me
> when things go south.
> 
> The only drawback is that by reporting the expected issues as passing it
> makes it too easy to forget about them and misinterpret the pass-rates.
> So this change modifies the junit backend to report the expected issues
> as skipped, making it more obvious when looking at the test graphs that
> these tests are not really passing, and that whatever functionality they
> target is not being fully covered.
> 
> This change also makes use of the junit `message` attribute to explain
> the reason of the skip.  (In fact, we could consider using the `message`
> attribute on other kind of failures to inform the piglit result, instead
> of using the non-standard `type`.)
> ---
>  framework/backends/junit.py | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/framework/backends/junit.py b/framework/backends/junit.py
> index 82f9c29..53b6086 100644
> --- a/framework/backends/junit.py
> +++ b/framework/backends/junit.py
> @@ -129,17 +129,19 @@ class JUnitBackend(FileBackend):
>              # Add relevant result value, if the result is pass then it doesn't
>              # need one of these statuses
>              if data['result'] == 'skip':
> -                etree.SubElement(element, 'skipped')
> +                res = etree.SubElement(element, 'skipped')
>  
>              elif data['result'] in ['warn', 'fail', 'dmesg-warn', 'dmesg-fail']:
>                  if expected_result == "failure":
>                      err.text += "\n\nWARN: passing test as an expected failure"
> +                    res = etree.SubElement(element, 'skipped', message='expected failure')
>                  else:
>                      res = etree.SubElement(element, 'failure')
>  
>              elif data['result'] == 'crash':
>                  if expected_result == "error":
>                      err.text += "\n\nWARN: passing test as an expected crash"
> +                    res = etree.SubElement(element, 'skipped', message='expected crash')
>                  else:
>                      res = etree.SubElement(element, 'error')
>  
> -- 
> 2.1.0
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: Digital signature
URL: <http://lists.freedesktop.org/archives/piglit/attachments/20150303/a097934e/attachment-0001.sig>