[Mesa-ci-intel] Clarification about 19.1 CI branches

[Clayton Craft]

On Wed, Jul 03, 2019 at 09:58:45AM +0200, Juan A. Suarez Romero wrote:
>On Tue, 2019-07-02 at 11:07 -0700, Clayton Craft wrote:
>> On Tue, Jul 02, 2019 at 06:19:10PM +0200, Juan A. Suarez Romero wrote:
>> > Hello.
>> >
>> > First of all, I'd like to know if staging/19.1 branch is not working properly in
>> > the CI.
>> >
>> > I've pushed several patches to the branch, but the CI keeps showing only the
>> > build from the 21th of last month:
>> >
>> > https://mesa-ci.01.org/mesa_19_1_staging/builds/
>>
>> There were some issues over the last few days where the CI results site was not
>> being updated (our IT did some 'upgrades' that broke things and I wasn't aware
>> of it until yesterday). Those issues have been resolved, and I have manually
>> triggered the import of the results for those two builds, they are visible now.
>>
>
>
>Now I got them. Thank you.
>
>> >
>> > The other clarification is about mesa_19_1 and mesa_19_1_daily.
>>
>> I've summarized the *current* stable release/staging CI jobs below (for my and
>> anyone else's benefit):
>>
>> mesa_19.1
>>         - triggers per-checkin
>>         - tests stable branch
>>         - does not test 32-bit and does not test a lot of older/slower platforms
>> mesa_19.1_daily
>>         - tests stable branch
>>         - tests 32-bit and a lot of older/slower platforms
>> mesa_19.1_staging
>>         - triggers per-checkin
>>         - tests staging branch
>>         - does not test 32-bit and does not test a lot of older/slower platforms
>> mesa_19.1_staging_weekly
>>         - triggered once a week on tuesday morning (PST)
>>         - tests staging branch
>>         - includes lots of older, slower hardware that would take too long to
>>           test per-commit
>>         - intended to help prevent regressions like the G33/i915_dri regression
>>           that was introduced on 19.0
>>
>
>
>I guess the staging_weekly is a new one you recently created. Never seen before.
>
>And neither seen mesa_19.1 (I only had 19.1_daily and 19.1_staging)
>
>> > The thing is that I only push into 19.1 just when I'm going to make the release
>> > pre-announcement. Which happens once every two weeks. I don't see the need to
>> > test 19.1 daily if it only will change once in two weeks.
>>
>> Yea this seems like it could be changed to where mesa_19.1 tests both m32, old
>> platforms, and atom platforms (in addition to the m64 and other platforms it
>> currently tests), and the daily stable branch job (mesa_19.1_daily) could be
>> removed.
>
>Yes, that's my proposal.
>
>Well, I need to clarify about the "push in 19.1 once per week".
>
>We make a push just before the pre-announcement, which is the Friday before the
>release day (every two weeks).
>
>But then in the release day, due the way releases are done, we need to do two
>pushes.
>
>
>A slightly different proposal could be to run only 32bit tests in mesa_19.1, as
>the 64bit tests are already tested with staging/19.1 (and I push the content of
>staging/19.1 into 19.1, so both branches are the same).
>
>> >
>> > The only interesting thing from this daily run is that it executes the 32bit
>> > testsuites. Something that mesa_19_1 does not do. So hence, when I push to 19.1,
>> > I need to wait for the next day if I want to see what happened with the 32bit
>> > testsuites.
>>
>> The intention of the mesa_19.1_staging_weekly CI job is to test 32-bit and
>> slow/old platforms (previously this was not happening on staging branch..), and
>> provide results before the release is made. It's a lot of load on the CI to do
>> this, hence the decision to make this job scheduled weekly instead of
>> per-checkin. Maybe weekly is too infrequent.
>>
>
>I see. Probably I would change the day of the run. We usually do the
>preannouncement on Friday and the release on Tuesday. Running the tests in
>Thursday (before the preannouncement) or in Monday (before the final release)
>would increase the chances of detecting any problem before.
>
>> >
>> > I think in the past we didn't have the daily setup, and the mesa_19_1 were
>> > running both the 64bit and the 32bit testsuites, after a push in 19.1 branch. I
>> > would like to bring this configuration, because we would saving time running the
>> > daily setup, and I would get the results for 32bits after pushing the 19.1
>> > release candidate.
>>
>> The reason 32-bit testing is deferred to run daily instead of per-checkin for
>> stable branch testing is because it generates a lot of load on CI: X number of
>> platforms doing both 32-bit *and* 64-bit runs of Y number of test suites.
>>
>> >
>> > What do you think?
>>
>> I think we need a better set of jobs that consider:
>>         1) coverage on 64-bit and 32-bit for both staging and stable branches
>>         2) coverage on all platforms for both staging and stable branches
>>         3) not overloading CI by testing too much too often
>>         4) getting you (release maintainer) feedback on staging branch/release
>>         candidates before stable release
>>
>
>We had created the staging branches as the place where we pushes quite
>frequently and want to do quick testing to detect any problem in advance.
>
>And the stable place is where we put all the commits in staging once we consider
>those should be the next release; so it is a more stable branch with less
>frequent pushes.
>
>So staging should do quick not very overloading CI, and stable do a deeper CI.
>Like you are proposing below :)
>
>> With these in mind, I propose the following set of CI jobs, replacing the
>> current set of CI jobs for staging/release branch testing:
>>
>> mesa_19.1
>>         - triggers per-checkin
>>         - tests stable branch
>>         - tests 32-bit and tests a lot of older/slower platforms
>> mesa_19.1_staging
>>         - triggers per-checkin
>>         - tests staging branch
>>         - does *not* test 32-bit and does *not* test a lot of older/slower platforms
>> mesa_19.1_staging_daily
>>         - triggered once per day
>>         - tests staging branch
>>         - tests 32-bit and tests a lot of older/slower platforms
>>         - intended to help prevent regressions like the G33/i915_dri regression
>>           that was introduced on 19.0 from being introduced into the stable
>>           branch
>>
>
>Sounds fantastic!
>
>> The stable release branch would get tested with everything CI can throw at it
>> when new commits are made.
>>
>> The staging branch would be tested with the same testing that happens on
>> developer branches today when new commits are made.
>>
>> The staging branch would also be tested daily with everything CI can throw at it
>> once per day, regardless of whether or not commits were made to it (the history
>> built up over time from this can also help identify intermittent failures).
>> Since this is a big job, running it per-checkin on the staging branch could lead
>> to the CI becoming overloaded easily if you, for example, pushed two changes to
>> the branch in a short amount of time.
>>
>
>
>I like the proposal. Thank you!
>
>	J.A.
>
>

Excellent, I have implemented the proposed changes, and will plan to do so for
future stable/staging branches (e.g. 19.2, etc) unless we decide to tweak it
some more :)

-Clayton
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/mesa-ci-intel/attachments/20190703/79f37f77/attachment.sig>