[Mesa-dev] so the development model is working?

[Alex Deucher]

On Sat, May 1, 2010 at 2:32 AM, Dave Airlie <airlied at gmail.com> wrote:
>>
>> I develop on multiple machines too, FWIW.  I do lots of ssh stuff like
>> Keith described.
>>
>> BTW, your last sentence can go either way.  From my point of view
>> it's: "Its a lot easier to just fix bugs on the 7.8 branch on those
>> machines, and when you get around to it later merging the changes back
>> into master, and dedicating a day to retesting master on as many
>> configurations as you think require it for you to be happy with the
>> work done."
>
> The thing is breaking master by accident or temporarily isn't near as
> bad as screwing up on the stable branch. I really want stable branch
> to be something I can pull a diff from at any point without worrying
> its broken stuff. I don't think we are good enough to stabilise a
> single driver or piece of mesa on the stable branch without causing
> collateral damage or unknown side effects in other drivers or parts of
> the codebase. Bugs are a natural side effect of any development, and
> if there are more people using master for development there is a
> chance of someone else tripping over the bugs, before hitting stable.
>
>>
>>
>>>> I'm usually working on master.  If I think I've found a bug (or
>>>> someone reports a bug in 7.8, for example) I just change terminal tabs
>>>> and start debugging in my 7.8 directory.  I don't have to save my work
>>>> on master or check out a different branch, etc.  Just switch to a
>>>> different terminal and fire up emacs.  After I fix the bug, I commit
>>>> it to the 7.8 branch and document it in the release notes file (I'm
>>>> practically the _only_ person who bothers to do that, btw!).  If I
>>>> think it's important to get the fix into master quickly (or there's
>>>> any doubt about applicability) I'll do a merge right away.  Otherwise,
>>>> I just resume whatever I was doing before, knowing that the fix will
>>>> propogate into master whenever we do a merge.
>>>
>>> Again I don't think is what the majority of developers at least the
>>> part-time ones do or anything close. They restrict their limited time
>>> to making master at any point in time work well for them. If you are
>>> developing a feature, and you fix a bug in a separate commit while
>>> working on the feature, taking a divergent path into 7.8 land is quite
>>> a detour, it usually involved git rebasing the stable 7.8 you checked
>>> out a month ago up to whatever is latest version, rebuilding it,
>>> applying your patch, building, testing, pushing, pulling into master,
>>> rebasing your current development work which may also need the bug
>>> fixed to continue on top of it. I really don't see this as a small
>>> divergence. If you are focused on doing development, this is a
>>> complete task switch, which could reliably be done in a separate task
>>> later, with arguably a lower chance of regressions.
>>
>> I think this is a matter of opinion because I think it's easy to
>> temporarily switch to my 7.8 tree when I find a bug while working on
>> master.  Also, if I'm working on file.c I don't have to temporarily
>> back-out my new work there in order to fix/commit a bug elsewhere in
>> the file.
>
> git stash is what usually works here, or git add -p, I generally just
> git add -p the fix, commit it to master, stash local changes, run
> test, push commit to master, unstash and keep going. I often queue up
> 2-3 fixes in my master before I finish the feature or bother testing
> all the fixes.
>
>>>> I think you exagerate how many testing steps you have to do for each
>>>> and every fix.  Absolutely, there are times when thorough testing is
>>>> needed.  But lots of simple bug fixes/changes in 7.8 can be merged
>>>> into master without worry.  I can provide many concrete examples if
>>>> you don't believe me.
>>>
>>> I can also provide quite a large long list of commits that aren't in
>>> 7.8 because nobody wants to deal with them. Also committing any fixes
>>> directly to a stable branch is something I've never really heard of as
>>> a standard. I would assume the expectation is that nobody willing
>>> introduces regressions, but lots of people accidentally do, and it
>>> would be better to find those in master first not in the stable branch
>>> when a distro has already shipped it.
>>
>> If you're concerned with causing regressions with a commit, you can do
>> several things including posting the patch for review first and/or
>> running a bunch of regression tests (which it sounds like you're
>> already doing) before committing.  Also, just because a bug fix works
>> on master is not a guarantee that it won't cause regressions on the
>> stable branch.  It goes both ways.
>
> The problem with regression tests is they don't cover enough hw
> combos. Like I can regression test some radeon easily, however when I
> shove stuff onto master, I get the side benefits of Intel testing for
> regressions in their driver, the other radeon guys doing tests on
> different hw combos etc.
>
>>> But they are stabilizing master for the next release. For r300 for
>>> example accel CopyTexImage was added in master, while fixing that
>>> Maciej also fixed a few FBO bugs. His focus was just on making master
>>> pass piglit better. He wasn't thinking about how applicable these
>>> fixes were to stable, he probably didn't consider the bugfixes to be
>>> stable material. Once master diverges from stable by more than a
>>> feature or two, most developers will need to work on master to make
>>> sure the features introduced are better tested, you don't want someone
>>> testing on 7.8 on the offchance that a feature in master will
>>> invalidate the tests by having replaced a codepath.
>>
>> I'd argue that stabilizing the stable branches is more important than
>> stabilizing master.  The release notes for a x.y.0 release clearly
>> state that it's a new development release and people especially
>> concerned with stability should stick with the previous x.(y-1).z
>> release.
>>
>> Plus, before a new x.y.0 development release is made, Ian's been
>> laying down a release timeframe that is meant for firming up the
>> release.  That's the time to try to really stabilize master.
>
> But I don't think that has happened at all, we've had a couple of
> releases with this development cycle and from what I can see all the
> developers are still developing on master.
>
>
>>>
>>>>                 Author             Signed- off-by
>>>> Brian Paul         20                    18
>>> Lots of gallium and mesa/st fixes - number of distros shipping
>>> anything out of here? 0.
>>>
>>>> Jeremy Huddleston  18                    17
>>> - apple builds fixes - 0 applicabilty
>>>
>>>> Ian Romanick        6                     1
>>> - version numbers + GLX fix - the only real fix for insanity that
>>> shouldn't have happened.
>>>
>>>> Dan Nicholson       4                     4
>>> gallium + egl + build fixes - 0 applicablilty no distro ships a 7.8.x
>>> gallium driver or EGL.
>>>
>>>> Vinson Lee          4                     0
>>> - 2 useful builds fixes - one i915g fix (anyone shipping it?)
>>>
>>>> Jesse Barnes        4                     0
>>> - some good DRI2 fixes, I'll allow Jesse to reveal his own opinions on
>>> getting those in.
>>>
>>>> Matthiew Herbb      4                     0
>>> gcc 3.3 build fixes - nice but nothing for a modern distro
>>>
>>>> Pierre Willenbrock  3                     0
>>> - useful fixes
>>>
>>>> Alex Deucher        2                     0
>>> - pci ids
>>>
>>>> Aaron Plattner      2                     0
>>> - useful fix
>>>
>>>> Chia-I Wu           2                     0
>>> - st/vega - does anyone ship it?
>>>
>>>> Maciej Cencora      2                     0
>>>> Michel Danzer       2                     0
>>>> Kristian Hogsberg   1                     0
>>>> Ben Skeggs          1                     0
>>>> Dave Arlie          0                     1
>>>> (plus a few other one-timers)
>>>
>>> So my guess here is that the majority of these things done on the 7.8
>>> branch have only a minor effect on the driver set that is currently
>>> being shipped by distros and the bugs people are seeing in the field.
>>> Do you at least agree with that?
>>
>> Yes, I do.  The point I was making is the development model I'm
>> following works pretty well.  Regardless of the contents or audience
>> of the 7.8.x branch, I feel each commit to the branch is a valid
>> fix/improvement.  And it's not painful to work in that manner.
>>
>> If I were working on hardware driver (such as I did with i965 last year)
>> I'd be following the same development model, making the i965 driver on
>> the 7.8 branch better and better.
>>
>>
>>
>>> Any gallium or egl related fix isn't
>>> useful to the consumers of this branch, other than I assume vmware
>>> consuming it. I'm also assuming you use the stable branch for stuff,
>>> again if I'm wrong please clarify.
>>>
>>>> If you're concerned about producing a stable driver, why aren't you
>>>> making more fixes to the 7.8/stable branch, whether by cherry picking
>>>> or whatever?  That's the whole point of it.  Master is not a stable
>>>> branch.
>>>
>>> Is cherry-picking not frowned upon?
>>
>> It is, but it hasn't stopped people.  After merging, those cherry
>> picks appear in the master history twice, unfortunately.  But I'd
>> rather have that than no fixes to the 7.8 branch.  I'm more concerned
>> with getting bug fixes into the stable branch than maintaining a
>> pristine git history.
>>
>>
>>
>>> I thought the whole point of the
>>> development model was that we don't cherry-pick stuff. If we are
>>> allowed to have two different development models, where one group
>>> cherry picks and one group pushes stuff to 7.8 and pulls, I think
>>> we'll end up with a lot worse conflicts. Cherry-picking to 7.8 and
>>> pulling back into master just seems to make a bad situation more
>>> painful. If you are entirely focused on 7.8 how do you know nobody
>>> hasn't already fixed something in master and just used the wrong
>>> development model..
>>
>> Scan/read the git commit emails.
>>
>>
>>> Hence why I think only the cherry-pick model
>>> actually works, and the non-cherry-picking model has just become
>>> something nobody is taking seriously at least with shipping code.
>>
>> Yeah, that's unfortunate.  If people would give it a try for a while
>> they'd get used to it and see it's not as hard as it's been made out
>> to be.  And again, periodically doing a merge is a lot quicker/easier
>> than cherry-picking oodles of individual commits.  I've tried both
>> ways - I know from experience.
>
>>>> Look above and see if you can guess why I prefer doing merges to
>>>> cherry-pick.?  I'd rather do 3 merges vs. 20+ cherry-picks.  Cherry
>>>> picking quickly becomes a PITA once you get beyond a handful of
>>>> patches or one commit per week or so.
>>>>
>
> Here's my other problem I didn't really clarify, from what I can see
> your model just doesn't scale well. If there are 15 developers working
> on this model, and only every 30-40 commits someone pulls 7.8 into
> master, they are going to get a bunch of conflicts they've no idea wtf
> to do with. So to make it useable for more than 1-2 developers, each
> developer needs to pull 7.8 into master after each set of fixes they
> do, if thats 1 fix or 50. Maybe if it doesn't conflict they can avoid
> pushing master, but if there is any hope of a conflict with master,
> they must push it. Now you end up with a merge:fix ratio of close to
> 1:1 and you really don't gain anything from this model. This works
> fine now because nobody is doing it, but I can't see how you think it
> will scale if everyone starts doing it, especially in areas where a
> few people work. I'm interested if you think this analysis is correct?
> maybe I'm missing something in how this scales. With your example
> there you'd end up doing closer to 10-15 merges. Not to mention the
> side effects Eric and Kristian pointed out with stable collateral
> regressions in master due to mismerges or simply being able to fix
> something nicer in master.
>
>
>>>
>>> The thing is I'm trying to take a step back from being a mesa
>>> developer to my other task which is being a mesa consumer and from
>>> what I can see I'm not getting the stuff to package that I get from
>>> various other projects, i.e. the kernel + X server/drivers. Whether
>>> that means we need a mesa stable maintainer and I should stand up and
>>> take on that task and only one person should be in charge of the
>>> stable branch might be something worth investigating.
>>>
>>> One idea I did have was that maybe people working on master could tag
>>> commits they'd like to see considered for stable with a line in the
>>> git commit, like the kernel has cc: stable at kernel.org. Then a stable
>>> maintainers can try pulling those back into stable, announce when they
>>> had something worth releasing, get everyone to start a test run on
>>> their hw and then release a final tarball, every 2-3 months.
>>
>> Getting everyone to do anything is hard.  And I don't know who has the
>> time to handle that role.  My work days at VMware are pretty full and
>> I don't have as much spare time for Mesa work as I did in the past.
>
> I'd be happy to try and take on some of this if people think its a
> good plan, i.e. if I can persuade people to tag commits in master with
> some common notation, it would be quite easy to automate
> cherry-picking them to stable and dropping any that didn't auto-pick.
> Its also quite hard when master and stable have diverged a bit in
> terms of features to know if the features you fix bugs in, even exist
> in master. I know at least for Maciej's r300 fixes a lot of the code
> in the fbo/texture code was different so he probably didn't suspect
> they code was in 7.8 in the first place. I know I've lost track when
> master was 2-3 months past 7.7 of what was in 7.7 and what wasn't. As
> gallium accelerates branch merges this isn't going to get easier.

I like this idea.  Perhaps we can do a stable nominations wiki page
like the xserver and the maintainer(s) of each section (core, intel,
amd, gallium, etc.) can review the nominations and other possible
candidates and cherry-pick back fixes periodically.  Or if we didn't
want to make it so formal, just have the maintainer(s) periodically
check for stuff to cherry-pick back.  I for one have tried to make a
conscious effort to get fixes back to the stable branches for areas
I'm familiar with.  However as Dave said, after a while it's hard to
remember what features are on what branches.  For some features, I
have to checkout the older branch and examine the code to see if the
bug fix is even relevant.

Alex