[Mesa-dev] so the development model is working?

[Dave Airlie]

>
>
> Dave,
>
> I'm sorry you're frustrated, but let's see if we can at least come to a
> better understanding of where we're each coming from.
>
> Your message seems to boil down to "cherry-pick fixes from master back
> to the stable branch" vs. "fix bugs in the stable branch and
> periodically merge to master".
>
> If you have issues with the timing of releases, take it up with Intel.
> They're the ones who suggested the quarterly release model.  I've been
> OK with that so far, but I don't think it's ideal.  Sometimes the
> quantity and collection of changes/fixes don't neatly fall into the
> calendar months.

I'm quite happy that we have timed based releases at least, since we
know when stuff needs to be stabilised and when we can get away with
not having master as good as it should be.

>
> I'd like to respond to some of your arguments:
>
> """So whats wrong with the current model. It basically seems to
>  suggest that everyone working on mesa should be working on the stable
>  branches and not mainline."""
>
> Of course not.  Many of us are working on new features or redesigning
> things.  That's what master (and feature branches) is for and is
> totally inappropriate for stable branches.
>
>
> """because if I add a new feature to my mainline
>  branch then happen to notice a bug fix along the way, then I have to
>  stop what I'm doing, check out stable, run a test suite, fix the bug
>  in stable, run another test suite, merge stable to master, run a test
>  suite on master before my fix, run a test suite on master after the
>  merge, hope I didn't break 5 other drivers doing the merge."""
>
> Hmmm, here's how I work:  I have several separate git check-outs in
> different directories.  At the moment I have four that I'm actively
> working on (plus ~30 inactive/older ones).  My active directories
> contain different branches (master vs. 7.8 vs. 7.7 etc) with different
> builds (swrast vs. DRI vs. llvm, etc).  I typically have a terminal/tab
> open on each of my active check-outs.  This lets me quickly and easily
> compare the results of Mesa 7.7 vs. 7.8 vs. master when I find a bug
> or want to compare performance, etc.  Do you really do all your work
> in one directory with just one check-out??

I happen to do mesa development work on about 10 different machines,
so yes I generally keep one mesa tree on each as close to master as I
can get. Again you are developing for swrast or for vmware. Try
developing for something like radeon and intel on different days, I
have to keep a number of test boxes with r100->r600 cards in them,
along with laptops at home where I idly do gallium work and also a
pile of laptops in the office. I don't do any mesa development on my
main workstation because I try to use it to read emails and stuff.  I
know you guys have worked on hw drivers in the past, but its always
been quite focused on one platform at a time. I don't get that luxury
working on a distro, the open bug list for 3D drivers on Fedora is
quite large and involves hopping machines and swapping hw
configurations quite a lot.

So when you are working on multiple hw drivers you don't really get
the liberty of just keeping a bunch of trees in one place that you
regularly up date and test. Its a lot easier to just work in master on
those machines, and when you get around to it later pulling the
changes back into stable, and dedicating a day to retesting stable on
as many configurations as you think require it for you to be happy
with the work done.


> I'm usually working on master.  If I think I've found a bug (or
> someone reports a bug in 7.8, for example) I just change terminal tabs
> and start debugging in my 7.8 directory.  I don't have to save my work
> on master or check out a different branch, etc.  Just switch to a
> different terminal and fire up emacs.  After I fix the bug, I commit
> it to the 7.8 branch and document it in the release notes file (I'm
> practically the _only_ person who bothers to do that, btw!).  If I
> think it's important to get the fix into master quickly (or there's
> any doubt about applicability) I'll do a merge right away.  Otherwise,
> I just resume whatever I was doing before, knowing that the fix will
> propogate into master whenever we do a merge.

Again I don't think is what the majority of developers at least the
part-time ones do or anything close. They restrict their limited time
to making master at any point in time work well for them. If you are
developing a feature, and you fix a bug in a separate commit while
working on the feature, taking a divergent path into 7.8 land is quite
a detour, it usually involved git rebasing the stable 7.8 you checked
out a month ago up to whatever is latest version, rebuilding it,
applying your patch, building, testing, pushing, pulling into master,
rebasing your current development work which may also need the bug
fixed to continue on top of it. I really don't see this as a small
divergence. If you are focused on doing development, this is a
complete task switch, which could reliably be done in a separate task
later, with arguably a lower chance of regressions.

>
> I think you exagerate how many testing steps you have to do for each
> and every fix.  Absolutely, there are times when thorough testing is
> needed.  But lots of simple bug fixes/changes in 7.8 can be merged
> into master without worry.  I can provide many concrete examples if
> you don't believe me.

I can also provide quite a large long list of commits that aren't in
7.8 because nobody wants to deal with them. Also committing any fixes
directly to a stable branch is something I've never really heard of as
a standard. I would assume the expectation is that nobody willing
introduces regressions, but lots of people accidentally do, and it
would be better to find those in master first not in the stable branch
when a distro has already shipped it.

> """So for instance I have 2-3 developers
>  who are stabilising master as hard as they can for the next release,
>  I'd like to out-of-line pull some of those fixes into stable and not
>  have the world end."""
>
> What is stabilizing?  It's finding/fixing bugs and running tests,
> right?  That's what the stable branch is all about so why not do that
> work on the stable branch from the get-go?

But they are stabilizing master for the next release. For r300 for
example accel CopyTexImage was added in master, while fixing that
Maciej also fixed a few FBO bugs. His focus was just on making master
pass piglit better. He wasn't thinking about how applicable these
fixes were to stable, he probably didn't consider the bugfixes to be
stable material. Once master diverges from stable by more than a
feature or two, most developers will need to work on master to make
sure the features introduced are better tested, you don't want someone
testing on 7.8 on the offchance that a feature in master will
invalidate the tests by having replaced a codepath.

> Master is the place where new work is done: new features,
> re-architecting, moving things around, etc.  Those changes (and new
> feature branches) can come along any time and potentially effect all
> the drivers and invalidate any testing done moments before.  Things
> get broke there all the time (that's inevitable - no matter how hard
> we try, nobody's a perfect coder).  Is that the place where you want
> your people trying to stabilize things and fix bugs?  Really?

For release yes. If we release a 7.8 tarball I'd like to have spent
the week or so before hand stabilise master. Maybe this is what is
missing from the process or maybe people are missing that we should
try and do this. A release tarball shouldn't be just master tarred up
and spat out. That is wrong and maybe I missed that is all people
wanted it to be. Stabilisation should happen on master yes. Controlled
destabilisation should happen on master, but not randomly at any time.

> BTW, let's take a look at the changes to the 7.8 branch.  There's
> approximately 85 commits on the 7.8 branch since it was created.
> Who's doing the work?  Let's see:
>
Let me break this list down a small bit then from the point of view of
a Linux distributor (or on behalf of all linux distros), who from what
I can see are the main people who want to consume a mesa 7.8.1
tarball.

>                 Author             Signed- off-by
> Brian Paul         20                    18
Lots of gallium and mesa/st fixes - number of distros shipping
anything out of here? 0.

> Jeremy Huddleston  18                    17
- apple builds fixes - 0 applicabilty

> Ian Romanick        6                     1
- version numbers + GLX fix - the only real fix for insanity that
shouldn't have happened.

> Dan Nicholson       4                     4
gallium + egl + build fixes - 0 applicablilty no distro ships a 7.8.x
gallium driver or EGL.

> Vinson Lee          4                     0
- 2 useful builds fixes - one i915g fix (anyone shipping it?)

> Jesse Barnes        4                     0
- some good DRI2 fixes, I'll allow Jesse to reveal his own opinions on
getting those in.

> Matthiew Herbb      4                     0
gcc 3.3 build fixes - nice but nothing for a modern distro

> Pierre Willenbrock  3                     0
- useful fixes

> Alex Deucher        2                     0
- pci ids

> Aaron Plattner      2                     0
- useful fix

> Chia-I Wu           2                     0
- st/vega - does anyone ship it?

> Maciej Cencora      2                     0
> Michel Danzer       2                     0
> Kristian Hogsberg   1                     0
> Ben Skeggs          1                     0
> Dave Arlie          0                     1
> (plus a few other one-timers)

So my guess here is that the majority of these things done on the 7.8
branch have only a minor effect on the driver set that is currently
being shipped by distros and the bugs people are seeing in the field.
Do you at least agree with that? Any gallium or egl related fix isn't
useful to the consumers of this branch, other than I assume vmware
consuming it. I'm also assuming you use the stable branch for stuff,
again if I'm wrong please clarify.

> If you're concerned about producing a stable driver, why aren't you
> making more fixes to the 7.8/stable branch, whether by cherry picking
> or whatever?  That's the whole point of it.  Master is not a stable
> branch.

Is cherry-picking not frowned upon? I thought the whole point of the
development model was that we don't cherry-pick stuff. If we are
allowed to have two different development models, where one group
cherry picks and one group pushes stuff to 7.8 and pulls, I think
we'll end up with a lot worse conflicts. Cherry-picking to 7.8 and
pulling back into master just seems to make a bad situation more
painful. If you are entirely focused on 7.8 how do you know nobody
hasn't already fixed something in master and just used the wrong
development model.. Hence why I think only the cherry-pick model
actually works, and the non-cherry-picking model has just become
something nobody is taking seriously at least with shipping code.

> Look above and see if you can guess why I prefer doing merges to
> cherry-pick.?  I'd rather do 3 merges vs. 20+ cherry-picks.  Cherry
> picking quickly becomes a PITA once you get beyond a handful of
> patches or one commit per week or so.
>
> Really, I get tired of hearing about the implied pain of fixing things
> in the stable branch when I'm the person doing most of the work to
> maintain that branch.

You are doing most of the work because nobody else can be bothered.
I'm actually really serious about this. From a distro POV only a few
things in the stable branch actually matter, (a) libGL + DRI
associated pieces, (b) core mesa + a few dri drivers (intel, and all
the radeon variants). Nobody is doing anything on these and I'm
calling out the development model for this, feel free to defend this,
but I'd also ask that anyone who agrees with my assessment, says so.

> Dave, I realize that you (and others) feel strongly about your way of
> doing things, but so do I.  Plus, I'm doing more to try to maintain a
> stable branch than anyone else.
>
> As the project founder/leader I walk a fine line between making rules
> and being flexible/open to accomodate others.  For example, as much as
> I wish Mesa had a uniform coding standard, I let people who develop
> various drivers use their own style since it's their body of work and
> I respect that.
>
> I think I'm a pretty easy-going project leader, I seldom, if ever,
> tell people what to do or not do (and never brow-beat or insult them
> as some other big project leaders do).  I don't want to tell you what
> to do in this case either.  You've done a lot of great work on Mesa
> over the years and I respect your opinion.  At this point, for now, I
> just hope you have a better understanding of where I'm coming from.
>
> I also hope the other developers will think more about getting bug
> fixes into the stable branch, one way or another, please.

The thing is I'm trying to take a step back from being a mesa
developer to my other task which is being a mesa consumer and from
what I can see I'm not getting the stuff to package that I get from
various other projects, i.e. the kernel + X server/drivers. Whether
that means we need a mesa stable maintainer and I should stand up and
take on that task and only one person should be in charge of the
stable branch might be something worth investigating.

One idea I did have was that maybe people working on master could tag
commits they'd like to see considered for stable with a line in the
git commit, like the kernel has cc: stable at kernel.org. Then a stable
maintainers can try pulling those back into stable, announce when they
had something worth releasing, get everyone to start a test run on
their hw and then release a final tarball, every 2-3 months.

I'd like to get more opinions from other developers before I broach
any more possible solutions. I know for a fact I'm not the only one
who has this problem, I'm just the only one who is stupid enough to
waste his time giving out about it.

Dave.