[Mesa-dev] How to merge Mesa changes which require corresponding piglit changes

[Mark Janes]

Michel Dänzer <michel at daenzer.net> writes:

> On 2019-11-15 4:02 p.m., Mark Janes wrote:
>> Michel Dänzer <michel at daenzer.net> writes:
>> 
>>> Now that the GitLab CI pipeline tests a snapshot of piglit with llvmpipe
>>> (https://gitlab.freedesktop.org/mesa/mesa/merge_requests/2468), the
>>> question has come up how to deal with inter-dependent Mesa/piglit
>>> changes (where merging only one or the other causes some piglit
>>> regressions).
>>>
>>>
>>> First of all, let it be clear that just merging the Mesa changes as-is
>>> and breaking the GitLab CI pipeline is not acceptable.
>>>
>>>
>>> From the Mesa POV, the easiest solution is:
>>>
>>> 1. Merge the piglit changes
>>> 2. In the Mesa MR (merge request), add a commit which updates piglit[0]
>>> 3. If the CI pipeline is green, the MR can be merged
>>>
>>>
>>> In case one wants to avoid alarms from external CI systems, another
>>> possibility is:
>> 
>> For the Intel CI, no alarm is generated if the piglit test is pushed
>> first.  Normal development process includes writing a piglit test to
>> illustrate the bug that is being fixed.
>
> Cool, but what if the piglit changes affect the results of existing
> tests? That was the situation yesterday which prompted this thread.

We attribute the status change to piglit in the CI config, within a few
hours.  The test shows up as a failure in CI until it is triaged.

>>> 1. In the Mesa MR, add a commit which disables the piglit tests broken
>>> by the Mesa changes.
>>> 2. If the CI pipeline is green, the MR can be merged
>>> 3. Merge the piglit changes
>>> 4. Create another Mesa MR which updates piglit[0] and re-enables the
>>> tests disabled in step 1
>>>
>>> I hope that covers it, don't hesitate to ask questions if something's
>>> still unclear.
>> 
>> It might help developers if CI generated the patch to make their pipeline
>> pass.
>
> It does for the test result list, if that's what you mean.
>
> However, that patch shouldn't be applied mechanically, but only after
> confirming that all changes in test results are expected. Ideally,
> whenever there are any new tests, the corresponding CI jobs should be
> run several times to make sure the new results are stable, otherwise any
> flaky tests should be excluded.
>
>
> -- 
> Earthling Michel Dänzer               |               https://redhat.com
> Libre software enthusiast             |             Mesa and X developer