[Piglit] [Bug 104700] New: Crashing tests with subtests no longer show in a list of regressions

[bugzilla-daemon at freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=104700

            Bug ID: 104700
           Summary: Crashing tests with subtests no longer show in a list
                    of regressions
           Product: piglit
           Version: unspecified
          Hardware: Other
                OS: All
            Status: NEW
          Severity: normal
          Priority: medium
         Component: infrastructure
          Assignee: fabianbieler at fastmail.fm
          Reporter: pavel.ondracka at email.cz
        QA Contact: piglit at lists.freedesktop.org

When some test with subtests starts crashing, it will not show in a piglit
summary as a regression, since the actual subtests will go from pass to notrun
and the failure will be only reported for the "unknown" test which will go from
notrun to crash, and hence neither will end in the list of regressions. It can
be still found in the changes summary, however the list of regressions is no
longer reliable due to this. I'm not sure if this is a bug or a feature,
however it seems somewhat unfortunate since this can lead to missed
regressions.

Was probably introduced by 
commit 938ec48e2575b78defd06d169f704ed8d4f11bce
Author: Fabian Bieler <fabianbieler at fastmail.fm>
Date:   Sat Jan 6 23:36:02 2018 +0100

    framework: Handle crashing subtest.

    Piglit silently ignored crashes in subtests.

    It's impossible to know what subtest crashed. Theoretically it might
    even be possible that a test crashes after the last call to
    piglit_report_subtest_result and thus no subtest crashed. Though the
    odds of that happening are probably pretty long.

    If a test with subtests crashes, this commit adds an additional subtest
    named "unknown" with result "crash".

    The issue that subsequent subtests are not run is not touched upon by
    this commit.

    This is more of a work-around. A proper fix would modify the C subtest
    framework to print a list of subtests it intends to run at the start of
    the test. This would enable the python subtest framework to know which
    subtest crashed.
    But that's a lot of work (it would require modifying more than 100
    tests) and this is better than nothing.

    See also: https://bugs.freedesktop.org/show_bug.cgi?id=74642

    Reviewed-by: Brian Paul <brianp at vmware.com>

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/piglit/attachments/20180119/809e1839/attachment.html>