[Piglit] [RFC] Piglit tags/releases

[Jordan Justen]

On Thu, Oct 2, 2014 at 4:43 PM, Emil Velikov <emil.l.velikov at gmail.com> wrote:
> On 02/10/14 21:44, Ian Romanick wrote:
>> On 09/30/2014 09:55 AM, Emil Velikov wrote:
>>> On 30/09/14 16:18, Ian Romanick wrote:
>>>> On 09/29/2014 10:01 AM, Matt Turner wrote:
>>>>> On Mon, Sep 29, 2014 at 9:34 AM, Emil Velikov <emil.l.velikov at gmail.com> wrote:
>>>>>>  - Something for our QA and other non-developer teams to cling onto.
>>>>>
>>>>> I think this has actually back fired for us when we tried. Ian
>>>>> probably remembers more.
>>>>
>>>> I agree with Matt.  The one thing that seems useful is having a tag to
>>>> mark the point in the piglit tree where a particular Mesa release was
>>>> tested.  The main I didn't do that on previous Mesa releases is that I
>>>> tested with my current work tree... which had a bunch of tests that
>>>> weren't upstream.  That would have made the tag be on a SHA1 that didn't
>>>> exist.
>>>>
>>>> Anything beyond that feels like wasting time catering to the wrong set
>>>> of users.
>>>>
>>> I must be missing something here - Matt says it backfired, but you say
>>> that you've not tagged your previous "mesa-xx-tested" because of local
>>> changes. How are those two related ?
>>>
>>> I'm not sure I see meaning behind such tag. Is your team (going to be)
>>> using it as a reference point of sorts ?
>>
>> I'm not sure what Matt meant about back firing.  Here's what I do know...
>>
>> We occasionally get bug reports from QA that a test has started failing
>> on a release branch, but the regression observed relative to the piglit
>> run of the previous release on that branch.  So, the test passed on Mesa
>> 37.5.1, but it now fails on a 37.5.2.  Since we don't know what piglit
>> was used on 37.5.1, we don't know whether the failure is due to a change
>> in Mesa or a change in piglit... and we can't trivally bisect piglit.
>>
>> Having a mesa-37.5.1-release-test tag enables us to sort that out quickly.
>>
> Indeed having a tag should help.
>
> Imho naming the tag explicitly against mesa sounds a bit backwards.
> Afaict the general consensus is - piglit is always paving, always stable
> test suite. Thus we check the product (mesa/etc.) *against* it - i.e.
> tested mesa X with piglit Y. Does this make sense ?

Emil, I would say that piglit is mainly driven by the needs of Mesa,
so it is useful to relate piglit "releases" to Mesa when possible.

Ian, all,

What about this compromise/monstrosity?
0.0.0-git20140911-1234abc-mesa-37.5.1

I also think this could work, as I'm not sure we need the date & sha1
in the tag:
0.0.0-tested-mesa-37.5.1

Finally, how about a tag of 0.0.0, with the "release" notes as the tag
comment documenting the Mesa release intended to be tested by the tag?

-Jordan