[Intel-gfx] [QUERY] How many CI mails is too many?

[Daniel Vetter]

On Tue, Nov 28, 2017 at 11:06 AM, Chris Wilson <chris at chris-wilson.co.uk> wrote:
> Quoting Joonas Lahtinen (2017-11-28 08:15:13)
>> On Mon, 2017-11-27 at 16:54 +0200, Arkadiusz Hiler wrote:
>> > Hey all,
>> >
>> > For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT
>> > results and "full IGT" results (if BAT has not failed).
>> >
>> > Recently we have added 32bit build check, and if that fails it sends out
>> > additional mail In-Reply-To the series.
>> >
>> > I am working on adding some static checks to the CI (spare and checkpatch at the
>> > moment, more may come in the future), which may generate even more commotion on
>> > the mailing list.
>> >
>> > How much of CI noise is too much and how you would like to have the results
>> > grouped?
>> >
>> > Couple of options to start the discussion:
>> >
>> >  1. Group all static checks (and the 32bit build?) into one mail:
>> >     - just one additional mail,
>> >     - may be hard to read in case of catastrophic failure,
>> >     - we can send it only when something actually fails.
>> >
>> >  2. Send out the results as a part of BAT results:
>> >     - even less noise than (1),
>> >     - BAT results already feel cluttered, this may decrease readability.
>> >
>> >  3. Have each check as a separate mail, but send it only if the check fails:
>> >     - noisy: may result in many mails, depending how many checks fail,
>> >     - easier to read and easier to follow on patchwork.
>>
>> The best user experience I could think of;
>>
>> 1. If all CI checks succeed, delay and only send one mail with all the
>> results. This would indicate it's good to merge, go do it.
>> 2. When a CI checks fail, immediately send that out so the developer
>> gets to work on the fix.
>>
>> Above requires that all the checks complete rather quickly and a trust
>> is gained to the system so that the absence of e-mail always means the
>> series is doing good, not that the system is clogged in some way :)
>
> Or just 2. The first being the compilation report; saying we
> have received your patch and it compiles fine, it will be queued to the
> farm currently in slot N (or it doesn't even compile!). The second being
> the success or failure of the CI run.
>
> From the user pov, we can't do anything until the CI report so
> intermediate emails saying congrats are just fluff. Useful simply to
> know the patch hasn't fall out of the system, but not supplying any
> actionable information.

BAT was meant to be that mail, with the added benefit that if a series
fails the basic sanity check you can ignore it for review and
everything. Still not quite there yet (and the recently undone change
of ratelimiting didn't help).
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch