<html>
<head>
<base href="https://bugs.freedesktop.org/">
</head>
<body><table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Bug ID</th>
<td><a class="bz_bug_link
bz_status_NEW "
title="NEW - Crashing tests with subtests no longer show in a list of regressions"
href="https://bugs.freedesktop.org/show_bug.cgi?id=104700">104700</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>Crashing tests with subtests no longer show in a list of regressions
</td>
</tr>
<tr>
<th>Product</th>
<td>piglit
</td>
</tr>
<tr>
<th>Version</th>
<td>unspecified
</td>
</tr>
<tr>
<th>Hardware</th>
<td>Other
</td>
</tr>
<tr>
<th>OS</th>
<td>All
</td>
</tr>
<tr>
<th>Status</th>
<td>NEW
</td>
</tr>
<tr>
<th>Severity</th>
<td>normal
</td>
</tr>
<tr>
<th>Priority</th>
<td>medium
</td>
</tr>
<tr>
<th>Component</th>
<td>infrastructure
</td>
</tr>
<tr>
<th>Assignee</th>
<td>fabianbieler@fastmail.fm
</td>
</tr>
<tr>
<th>Reporter</th>
<td>pavel.ondracka@email.cz
</td>
</tr>
<tr>
<th>QA Contact</th>
<td>piglit@lists.freedesktop.org
</td>
</tr></table>
<p>
<div>
<pre>When some test with subtests starts crashing, it will not show in a piglit
summary as a regression, since the actual subtests will go from pass to notrun
and the failure will be only reported for the "unknown" test which will go from
notrun to crash, and hence neither will end in the list of regressions. It can
be still found in the changes summary, however the list of regressions is no
longer reliable due to this. I'm not sure if this is a bug or a feature,
however it seems somewhat unfortunate since this can lead to missed
regressions.
Was probably introduced by
commit 938ec48e2575b78defd06d169f704ed8d4f11bce
Author: Fabian Bieler <<a href="mailto:fabianbieler@fastmail.fm">fabianbieler@fastmail.fm</a>>
Date: Sat Jan 6 23:36:02 2018 +0100
framework: Handle crashing subtest.
Piglit silently ignored crashes in subtests.
It's impossible to know what subtest crashed. Theoretically it might
even be possible that a test crashes after the last call to
piglit_report_subtest_result and thus no subtest crashed. Though the
odds of that happening are probably pretty long.
If a test with subtests crashes, this commit adds an additional subtest
named "unknown" with result "crash".
The issue that subsequent subtests are not run is not touched upon by
this commit.
This is more of a work-around. A proper fix would modify the C subtest
framework to print a list of subtests it intends to run at the start of
the test. This would enable the python subtest framework to know which
subtest crashed.
But that's a lot of work (it would require modifying more than 100
tests) and this is better than nothing.
See also: <a class="bz_bug_link
bz_status_NEW "
title="NEW - crashes during subtests are unreported, as are all subsequent subtests in the same test"
href="show_bug.cgi?id=74642">https://bugs.freedesktop.org/show_bug.cgi?id=74642</a>
Reviewed-by: Brian Paul <<a href="mailto:brianp@vmware.com">brianp@vmware.com</a>></pre>
</div>
</p>
<hr>
<span>You are receiving this mail because:</span>
<ul>
<li>You are the QA Contact for the bug.</li>
</ul>
</body>
</html>