[Mesa-dev] How to merge Mesa changes which require corresponding piglit changes

[Michel Dänzer]

On 2019-11-15 4:02 p.m., Mark Janes wrote:
> Michel Dänzer <michel at daenzer.net> writes:
> 
>> Now that the GitLab CI pipeline tests a snapshot of piglit with llvmpipe
>> (https://gitlab.freedesktop.org/mesa/mesa/merge_requests/2468), the
>> question has come up how to deal with inter-dependent Mesa/piglit
>> changes (where merging only one or the other causes some piglit
>> regressions).
>>
>>
>> First of all, let it be clear that just merging the Mesa changes as-is
>> and breaking the GitLab CI pipeline is not acceptable.
>>
>>
>> From the Mesa POV, the easiest solution is:
>>
>> 1. Merge the piglit changes
>> 2. In the Mesa MR (merge request), add a commit which updates piglit[0]
>> 3. If the CI pipeline is green, the MR can be merged
>>
>>
>> In case one wants to avoid alarms from external CI systems, another
>> possibility is:
> 
> For the Intel CI, no alarm is generated if the piglit test is pushed
> first.  Normal development process includes writing a piglit test to
> illustrate the bug that is being fixed.

Cool, but what if the piglit changes affect the results of existing
tests? That was the situation yesterday which prompted this thread.


>> 1. In the Mesa MR, add a commit which disables the piglit tests broken
>> by the Mesa changes.
>> 2. If the CI pipeline is green, the MR can be merged
>> 3. Merge the piglit changes
>> 4. Create another Mesa MR which updates piglit[0] and re-enables the
>> tests disabled in step 1
>>
>> I hope that covers it, don't hesitate to ask questions if something's
>> still unclear.
> 
> It might help developers if CI generated the patch to make their pipeline
> pass.

It does for the test result list, if that's what you mean.

However, that patch shouldn't be applied mechanically, but only after
confirming that all changes in test results are expected. Ideally,
whenever there are any new tests, the corresponding CI jobs should be
run several times to make sure the new results are stable, otherwise any
flaky tests should be excluded.


-- 
Earthling Michel Dänzer               |               https://redhat.com
Libre software enthusiast             |             Mesa and X developer