[Piglit] [PATCH 1/2] framework: change 'regressions' to mean 'regressions in the last test run'

[Ilia Mirkin]

On Mon, Oct 24, 2016 at 11:04 AM, Nicolai Hähnle <nhaehnle at gmail.com> wrote:
> On 24.10.2016 16:57, Ilia Mirkin wrote:
>>
>> So is the idea that now regressions shows regressions compared to an
>> initial "golden" run?
>
>
> No, it's the dual of that: the latest run is compared to everything before,
> i.e. it answers the question: what's failing _now_ that used to work at some
> point in the past?
>
>
>> I don't like the idea of only changing regressions. These things were
>> previously pretty consistent...
>>
>> fixes + regressions + new + disabled = changes
>
>
> But this "sums" isn't disjoint: a test can appear in all of fixes,
> regressions, and new (and in some unusual cases even disabled)
> simultaneously. That makes the status quo pretty useless IMO.
>
> Maybe 'fixes' should get a similar do-over to 'regressions', but I'm not
> sure what that would look like.
>
> Let me turn this around and ask: (a) are you using the regressions page
> today, and (b) if so, how? Also, the same questions for the fixes page,
> because I have no use for that at all :)

I have two disjoint use-cases, neither of which will break as a result
of your changes:

(a) Put up a bunch of unrelated runs. In that case, only the problems
page really matters.

(b) Put up two runs, compare a new thing to an old thing. In that
case, I will generally look at the changes page, unless that thing is
massively polluted by a silly rename, in which case I'll look at fixes
+ regressions and disregard the new/disabled. With a scan over the
problems page in case anything obvious pops up on there.

However there's something to be said for logical consistency. Each
"diff" page working in the same way makes sense. Why is regressions
different from fixes?

  -ilia