[Piglit] [RFC] Piglit tags/releases

[Emil Velikov]

On 02/10/14 21:44, Ian Romanick wrote:
> On 09/30/2014 09:55 AM, Emil Velikov wrote:
>> On 30/09/14 16:18, Ian Romanick wrote:
>>> On 09/29/2014 10:01 AM, Matt Turner wrote:
>>>> On Mon, Sep 29, 2014 at 9:34 AM, Emil Velikov <emil.l.velikov at gmail.com> wrote:
>>>>> While I've been through the RELEASES document I believe it would be
>>>>> beneficial if we regularly create a tag("release"), that is to serve the
>>>>> following
>>>>>  - Human understandable format
>>>>> I.e. version 1.0.2 comes after 1.0.1, oh there is even date in there.
>>>>
>>>> See next point.
>>>>
>>>>>  - Something everyone can parse, unlike b33979a8f5c852fbffc072b0.
>>>>> When you don't have the tree at hand or don't know what git is.
>>>>
>>>> If either of these is the case, you have no business with piglit.
>>>>
>>>>>  - Ease distributions interested in packaging piglit.
>>>>
>>>> I don't see value in distributions packaging piglit.
>>>>
>>>>>  - Something for our QA and other non-developer teams to cling onto.
>>>>
>>>> I think this has actually back fired for us when we tried. Ian
>>>> probably remembers more.
>>>
>>> I agree with Matt.  The one thing that seems useful is having a tag to
>>> mark the point in the piglit tree where a particular Mesa release was
>>> tested.  The main I didn't do that on previous Mesa releases is that I
>>> tested with my current work tree... which had a bunch of tests that
>>> weren't upstream.  That would have made the tag be on a SHA1 that didn't
>>> exist.
>>>
>>> Anything beyond that feels like wasting time catering to the wrong set
>>> of users.
>>>
>> I must be missing something here - Matt says it backfired, but you say
>> that you've not tagged your previous "mesa-xx-tested" because of local
>> changes. How are those two related ?
>>
>> I'm not sure I see meaning behind such tag. Is your team (going to be)
>> using it as a reference point of sorts ?
> 
> I'm not sure what Matt meant about back firing.  Here's what I do know...
> 
> We occasionally get bug reports from QA that a test has started failing
> on a release branch, but the regression observed relative to the piglit
> run of the previous release on that branch.  So, the test passed on Mesa
> 37.5.1, but it now fails on a 37.5.2.  Since we don't know what piglit
> was used on 37.5.1, we don't know whether the failure is due to a change
> in Mesa or a change in piglit... and we can't trivally bisect piglit.
> 
> Having a mesa-37.5.1-release-test tag enables us to sort that out quickly.
> 
Indeed having a tag should help.

Imho naming the tag explicitly against mesa sounds a bit backwards.
Afaict the general consensus is - piglit is always paving, always stable
test suite. Thus we check the product (mesa/etc.) *against* it - i.e.
tested mesa X with piglit Y. Does this make sense ?

-Emil

>> Thanks
>> Emil
>