[Mesa-ci-intel] Clarification about 19.1 CI branches

[Juan A. Suarez Romero]

On Tue, 2019-07-02 at 11:07 -0700, Clayton Craft wrote:
> On Tue, Jul 02, 2019 at 06:19:10PM +0200, Juan A. Suarez Romero wrote:
> > Hello.
> > 
> > First of all, I'd like to know if staging/19.1 branch is not working properly in
> > the CI.
> > 
> > I've pushed several patches to the branch, but the CI keeps showing only the
> > build from the 21th of last month:
> > 
> > https://mesa-ci.01.org/mesa_19_1_staging/builds/
> 
> There were some issues over the last few days where the CI results site was not
> being updated (our IT did some 'upgrades' that broke things and I wasn't aware
> of it until yesterday). Those issues have been resolved, and I have manually
> triggered the import of the results for those two builds, they are visible now.
> 


Now I got them. Thank you.

> > 
> > The other clarification is about mesa_19_1 and mesa_19_1_daily.
> 
> I've summarized the *current* stable release/staging CI jobs below (for my and
> anyone else's benefit):
> 
> mesa_19.1
>         - triggers per-checkin
>         - tests stable branch
>         - does not test 32-bit and does not test a lot of older/slower platforms
> mesa_19.1_daily
>         - tests stable branch
>         - tests 32-bit and a lot of older/slower platforms
> mesa_19.1_staging
>         - triggers per-checkin
>         - tests staging branch
>         - does not test 32-bit and does not test a lot of older/slower platforms
> mesa_19.1_staging_weekly
>         - triggered once a week on tuesday morning (PST)
>         - tests staging branch
>         - includes lots of older, slower hardware that would take too long to
>           test per-commit
>         - intended to help prevent regressions like the G33/i915_dri regression
>           that was introduced on 19.0
> 


I guess the staging_weekly is a new one you recently created. Never seen before.

And neither seen mesa_19.1 (I only had 19.1_daily and 19.1_staging)

> > The thing is that I only push into 19.1 just when I'm going to make the release
> > pre-announcement. Which happens once every two weeks. I don't see the need to
> > test 19.1 daily if it only will change once in two weeks.
> 
> Yea this seems like it could be changed to where mesa_19.1 tests both m32, old
> platforms, and atom platforms (in addition to the m64 and other platforms it
> currently tests), and the daily stable branch job (mesa_19.1_daily) could be
> removed.

Yes, that's my proposal.

Well, I need to clarify about the "push in 19.1 once per week".

We make a push just before the pre-announcement, which is the Friday before the
release day (every two weeks).

But then in the release day, due the way releases are done, we need to do two
pushes.


A slightly different proposal could be to run only 32bit tests in mesa_19.1, as
the 64bit tests are already tested with staging/19.1 (and I push the content of
staging/19.1 into 19.1, so both branches are the same).

> > 
> > The only interesting thing from this daily run is that it executes the 32bit
> > testsuites. Something that mesa_19_1 does not do. So hence, when I push to 19.1,
> > I need to wait for the next day if I want to see what happened with the 32bit
> > testsuites.
> 
> The intention of the mesa_19.1_staging_weekly CI job is to test 32-bit and
> slow/old platforms (previously this was not happening on staging branch..), and
> provide results before the release is made. It's a lot of load on the CI to do
> this, hence the decision to make this job scheduled weekly instead of
> per-checkin. Maybe weekly is too infrequent.
> 

I see. Probably I would change the day of the run. We usually do the
preannouncement on Friday and the release on Tuesday. Running the tests in
Thursday (before the preannouncement) or in Monday (before the final release)
would increase the chances of detecting any problem before.

> > 
> > I think in the past we didn't have the daily setup, and the mesa_19_1 were
> > running both the 64bit and the 32bit testsuites, after a push in 19.1 branch. I
> > would like to bring this configuration, because we would saving time running the
> > daily setup, and I would get the results for 32bits after pushing the 19.1
> > release candidate.
> 
> The reason 32-bit testing is deferred to run daily instead of per-checkin for
> stable branch testing is because it generates a lot of load on CI: X number of
> platforms doing both 32-bit *and* 64-bit runs of Y number of test suites.
> 
> > 
> > What do you think?
> 
> I think we need a better set of jobs that consider:
>         1) coverage on 64-bit and 32-bit for both staging and stable branches
>         2) coverage on all platforms for both staging and stable branches
>         3) not overloading CI by testing too much too often
>         4) getting you (release maintainer) feedback on staging branch/release
>         candidates before stable release
> 

We had created the staging branches as the place where we pushes quite
frequently and want to do quick testing to detect any problem in advance.

And the stable place is where we put all the commits in staging once we consider
those should be the next release; so it is a more stable branch with less
frequent pushes.

So staging should do quick not very overloading CI, and stable do a deeper CI.
Like you are proposing below :)

> With these in mind, I propose the following set of CI jobs, replacing the
> current set of CI jobs for staging/release branch testing:
> 
> mesa_19.1
>         - triggers per-checkin
>         - tests stable branch
>         - tests 32-bit and tests a lot of older/slower platforms
> mesa_19.1_staging
>         - triggers per-checkin
>         - tests staging branch
>         - does *not* test 32-bit and does *not* test a lot of older/slower platforms
> mesa_19.1_staging_daily
>         - triggered once per day
>         - tests staging branch
>         - tests 32-bit and tests a lot of older/slower platforms
>         - intended to help prevent regressions like the G33/i915_dri regression
>           that was introduced on 19.0 from being introduced into the stable
>           branch
> 

Sounds fantastic!

> The stable release branch would get tested with everything CI can throw at it
> when new commits are made.
> 
> The staging branch would be tested with the same testing that happens on
> developer branches today when new commits are made.
> 
> The staging branch would also be tested daily with everything CI can throw at it
> once per day, regardless of whether or not commits were made to it (the history
> built up over time from this can also help identify intermittent failures).
> Since this is a big job, running it per-checkin on the staging branch could lead
> to the CI becoming overloaded easily if you, for example, pushed two changes to
> the branch in a short amount of time.
> 


I like the proposal. Thank you!

	J.A.