[Mesa-dev] so the development model is working?

[Jakob Bornecrantz]

On Sat, May 1, 2010 at 3:00 PM, Alex Deucher <alexdeucher at gmail.com> wrote:
> On Sat, May 1, 2010 at 2:32 AM, Dave Airlie <airlied at gmail.com> wrote:
>>>
>>> I develop on multiple machines too, FWIW.  I do lots of ssh stuff like
>>> Keith described.
>>>
>>> BTW, your last sentence can go either way.  From my point of view
>>> it's: "Its a lot easier to just fix bugs on the 7.8 branch on those
>>> machines, and when you get around to it later merging the changes back
>>> into master, and dedicating a day to retesting master on as many
>>> configurations as you think require it for you to be happy with the
>>> work done."
>>
>> The thing is breaking master by accident or temporarily isn't near as
>> bad as screwing up on the stable branch. I really want stable branch
>> to be something I can pull a diff from at any point without worrying
>> its broken stuff. I don't think we are good enough to stabilise a
>> single driver or piece of mesa on the stable branch without causing
>> collateral damage or unknown side effects in other drivers or parts of
>> the codebase. Bugs are a natural side effect of any development, and
>> if there are more people using master for development there is a
>> chance of someone else tripping over the bugs, before hitting stable.
>>
>>>
>>>
>>>>> I'm usually working on master.  If I think I've found a bug (or
>>>>> someone reports a bug in 7.8, for example) I just change terminal tabs
>>>>> and start debugging in my 7.8 directory.  I don't have to save my work
>>>>> on master or check out a different branch, etc.  Just switch to a
>>>>> different terminal and fire up emacs.  After I fix the bug, I commit
>>>>> it to the 7.8 branch and document it in the release notes file (I'm
>>>>> practically the _only_ person who bothers to do that, btw!).  If I
>>>>> think it's important to get the fix into master quickly (or there's
>>>>> any doubt about applicability) I'll do a merge right away.  Otherwise,
>>>>> I just resume whatever I was doing before, knowing that the fix will
>>>>> propogate into master whenever we do a merge.
>>>>
>>>> Again I don't think is what the majority of developers at least the
>>>> part-time ones do or anything close. They restrict their limited time
>>>> to making master at any point in time work well for them. If you are
>>>> developing a feature, and you fix a bug in a separate commit while
>>>> working on the feature, taking a divergent path into 7.8 land is quite
>>>> a detour, it usually involved git rebasing the stable 7.8 you checked
>>>> out a month ago up to whatever is latest version, rebuilding it,
>>>> applying your patch, building, testing, pushing, pulling into master,
>>>> rebasing your current development work which may also need the bug
>>>> fixed to continue on top of it. I really don't see this as a small
>>>> divergence. If you are focused on doing development, this is a
>>>> complete task switch, which could reliably be done in a separate task
>>>> later, with arguably a lower chance of regressions.
>>>
>>> I think this is a matter of opinion because I think it's easy to
>>> temporarily switch to my 7.8 tree when I find a bug while working on
>>> master.  Also, if I'm working on file.c I don't have to temporarily
>>> back-out my new work there in order to fix/commit a bug elsewhere in
>>> the file.
>>
>> git stash is what usually works here, or git add -p, I generally just
>> git add -p the fix, commit it to master, stash local changes, run
>> test, push commit to master, unstash and keep going. I often queue up
>> 2-3 fixes in my master before I finish the feature or bother testing
>> all the fixes.
>>
>>>>> I think you exagerate how many testing steps you have to do for each
>>>>> and every fix.  Absolutely, there are times when thorough testing is
>>>>> needed.  But lots of simple bug fixes/changes in 7.8 can be merged
>>>>> into master without worry.  I can provide many concrete examples if
>>>>> you don't believe me.
>>>>
>>>> I can also provide quite a large long list of commits that aren't in
>>>> 7.8 because nobody wants to deal with them. Also committing any fixes
>>>> directly to a stable branch is something I've never really heard of as
>>>> a standard. I would assume the expectation is that nobody willing
>>>> introduces regressions, but lots of people accidentally do, and it
>>>> would be better to find those in master first not in the stable branch
>>>> when a distro has already shipped it.
>>>
>>> If you're concerned with causing regressions with a commit, you can do
>>> several things including posting the patch for review first and/or
>>> running a bunch of regression tests (which it sounds like you're
>>> already doing) before committing.  Also, just because a bug fix works
>>> on master is not a guarantee that it won't cause regressions on the
>>> stable branch.  It goes both ways.
>>
>> The problem with regression tests is they don't cover enough hw
>> combos. Like I can regression test some radeon easily, however when I
>> shove stuff onto master, I get the side benefits of Intel testing for
>> regressions in their driver, the other radeon guys doing tests on
>> different hw combos etc.
>>
>>>> But they are stabilizing master for the next release. For r300 for
>>>> example accel CopyTexImage was added in master, while fixing that
>>>> Maciej also fixed a few FBO bugs. His focus was just on making master
>>>> pass piglit better. He wasn't thinking about how applicable these
>>>> fixes were to stable, he probably didn't consider the bugfixes to be
>>>> stable material. Once master diverges from stable by more than a
>>>> feature or two, most developers will need to work on master to make
>>>> sure the features introduced are better tested, you don't want someone
>>>> testing on 7.8 on the offchance that a feature in master will
>>>> invalidate the tests by having replaced a codepath.
>>>
>>> I'd argue that stabilizing the stable branches is more important than
>>> stabilizing master.  The release notes for a x.y.0 release clearly
>>> state that it's a new development release and people especially
>>> concerned with stability should stick with the previous x.(y-1).z
>>> release.
>>>
>>> Plus, before a new x.y.0 development release is made, Ian's been
>>> laying down a release timeframe that is meant for firming up the
>>> release.  That's the time to try to really stabilize master.
>>
>> But I don't think that has happened at all, we've had a couple of
>> releases with this development cycle and from what I can see all the
>> developers are still developing on master.
>>
>>
>>>>
>>>>>                 Author             Signed- off-by
>>>>> Brian Paul         20                    18
>>>> Lots of gallium and mesa/st fixes - number of distros shipping
>>>> anything out of here? 0.
>>>>
>>>>> Jeremy Huddleston  18                    17
>>>> - apple builds fixes - 0 applicabilty
>>>>
>>>>> Ian Romanick        6                     1
>>>> - version numbers + GLX fix - the only real fix for insanity that
>>>> shouldn't have happened.
>>>>
>>>>> Dan Nicholson       4                     4
>>>> gallium + egl + build fixes - 0 applicablilty no distro ships a 7.8.x
>>>> gallium driver or EGL.
>>>>
>>>>> Vinson Lee          4                     0
>>>> - 2 useful builds fixes - one i915g fix (anyone shipping it?)
>>>>
>>>>> Jesse Barnes        4                     0
>>>> - some good DRI2 fixes, I'll allow Jesse to reveal his own opinions on
>>>> getting those in.
>>>>
>>>>> Matthiew Herbb      4                     0
>>>> gcc 3.3 build fixes - nice but nothing for a modern distro
>>>>
>>>>> Pierre Willenbrock  3                     0
>>>> - useful fixes
>>>>
>>>>> Alex Deucher        2                     0
>>>> - pci ids
>>>>
>>>>> Aaron Plattner      2                     0
>>>> - useful fix
>>>>
>>>>> Chia-I Wu           2                     0
>>>> - st/vega - does anyone ship it?
>>>>
>>>>> Maciej Cencora      2                     0
>>>>> Michel Danzer       2                     0
>>>>> Kristian Hogsberg   1                     0
>>>>> Ben Skeggs          1                     0
>>>>> Dave Arlie          0                     1
>>>>> (plus a few other one-timers)
>>>>
>>>> So my guess here is that the majority of these things done on the 7.8
>>>> branch have only a minor effect on the driver set that is currently
>>>> being shipped by distros and the bugs people are seeing in the field.
>>>> Do you at least agree with that?
>>>
>>> Yes, I do.  The point I was making is the development model I'm
>>> following works pretty well.  Regardless of the contents or audience
>>> of the 7.8.x branch, I feel each commit to the branch is a valid
>>> fix/improvement.  And it's not painful to work in that manner.
>>>
>>> If I were working on hardware driver (such as I did with i965 last year)
>>> I'd be following the same development model, making the i965 driver on
>>> the 7.8 branch better and better.
>>>
>>>
>>>
>>>> Any gallium or egl related fix isn't
>>>> useful to the consumers of this branch, other than I assume vmware
>>>> consuming it. I'm also assuming you use the stable branch for stuff,
>>>> again if I'm wrong please clarify.
>>>>
>>>>> If you're concerned about producing a stable driver, why aren't you
>>>>> making more fixes to the 7.8/stable branch, whether by cherry picking
>>>>> or whatever?  That's the whole point of it.  Master is not a stable
>>>>> branch.
>>>>
>>>> Is cherry-picking not frowned upon?
>>>
>>> It is, but it hasn't stopped people.  After merging, those cherry
>>> picks appear in the master history twice, unfortunately.  But I'd
>>> rather have that than no fixes to the 7.8 branch.  I'm more concerned
>>> with getting bug fixes into the stable branch than maintaining a
>>> pristine git history.
>>>
>>>
>>>
>>>> I thought the whole point of the
>>>> development model was that we don't cherry-pick stuff. If we are
>>>> allowed to have two different development models, where one group
>>>> cherry picks and one group pushes stuff to 7.8 and pulls, I think
>>>> we'll end up with a lot worse conflicts. Cherry-picking to 7.8 and
>>>> pulling back into master just seems to make a bad situation more
>>>> painful. If you are entirely focused on 7.8 how do you know nobody
>>>> hasn't already fixed something in master and just used the wrong
>>>> development model..
>>>
>>> Scan/read the git commit emails.
>>>
>>>
>>>> Hence why I think only the cherry-pick model
>>>> actually works, and the non-cherry-picking model has just become
>>>> something nobody is taking seriously at least with shipping code.
>>>
>>> Yeah, that's unfortunate.  If people would give it a try for a while
>>> they'd get used to it and see it's not as hard as it's been made out
>>> to be.  And again, periodically doing a merge is a lot quicker/easier
>>> than cherry-picking oodles of individual commits.  I've tried both
>>> ways - I know from experience.
>>
>>>>> Look above and see if you can guess why I prefer doing merges to
>>>>> cherry-pick.?  I'd rather do 3 merges vs. 20+ cherry-picks.  Cherry
>>>>> picking quickly becomes a PITA once you get beyond a handful of
>>>>> patches or one commit per week or so.
>>>>>
>>
>> Here's my other problem I didn't really clarify, from what I can see
>> your model just doesn't scale well. If there are 15 developers working
>> on this model, and only every 30-40 commits someone pulls 7.8 into
>> master, they are going to get a bunch of conflicts they've no idea wtf
>> to do with. So to make it useable for more than 1-2 developers, each
>> developer needs to pull 7.8 into master after each set of fixes they
>> do, if thats 1 fix or 50. Maybe if it doesn't conflict they can avoid
>> pushing master, but if there is any hope of a conflict with master,
>> they must push it. Now you end up with a merge:fix ratio of close to
>> 1:1 and you really don't gain anything from this model. This works
>> fine now because nobody is doing it, but I can't see how you think it
>> will scale if everyone starts doing it, especially in areas where a
>> few people work. I'm interested if you think this analysis is correct?
>> maybe I'm missing something in how this scales. With your example
>> there you'd end up doing closer to 10-15 merges. Not to mention the
>> side effects Eric and Kristian pointed out with stable collateral
>> regressions in master due to mismerges or simply being able to fix
>> something nicer in master.
>>
>>
>>>>
>>>> The thing is I'm trying to take a step back from being a mesa
>>>> developer to my other task which is being a mesa consumer and from
>>>> what I can see I'm not getting the stuff to package that I get from
>>>> various other projects, i.e. the kernel + X server/drivers. Whether
>>>> that means we need a mesa stable maintainer and I should stand up and
>>>> take on that task and only one person should be in charge of the
>>>> stable branch might be something worth investigating.
>>>>
>>>> One idea I did have was that maybe people working on master could tag
>>>> commits they'd like to see considered for stable with a line in the
>>>> git commit, like the kernel has cc: stable at kernel.org. Then a stable
>>>> maintainers can try pulling those back into stable, announce when they
>>>> had something worth releasing, get everyone to start a test run on
>>>> their hw and then release a final tarball, every 2-3 months.
>>>
>>> Getting everyone to do anything is hard.  And I don't know who has the
>>> time to handle that role.  My work days at VMware are pretty full and
>>> I don't have as much spare time for Mesa work as I did in the past.
>>
>> I'd be happy to try and take on some of this if people think its a
>> good plan, i.e. if I can persuade people to tag commits in master with
>> some common notation, it would be quite easy to automate
>> cherry-picking them to stable and dropping any that didn't auto-pick.
>> Its also quite hard when master and stable have diverged a bit in
>> terms of features to know if the features you fix bugs in, even exist
>> in master. I know at least for Maciej's r300 fixes a lot of the code
>> in the fbo/texture code was different so he probably didn't suspect
>> they code was in 7.8 in the first place. I know I've lost track when
>> master was 2-3 months past 7.7 of what was in 7.7 and what wasn't. As
>> gallium accelerates branch merges this isn't going to get easier.
>
> I like this idea.  Perhaps we can do a stable nominations wiki page
> like the xserver and the maintainer(s) of each section (core, intel,
> amd, gallium, etc.) can review the nominations and other possible
> candidates and cherry-pick back fixes periodically.  Or if we didn't
> want to make it so formal, just have the maintainer(s) periodically
> check for stuff to cherry-pick back.  I for one have tried to make a
> conscious effort to get fixes back to the stable branches for areas
> I'm familiar with.  However as Dave said, after a while it's hard to
> remember what features are on what branches.  For some features, I
> have to checkout the older branch and examine the code to see if the
> bug fix is even relevant.

My thought about this is that this idea adds more work for all
developers, not only does the developer have to consider if a patch is
suitable for stable but they also need to update a wiki and a such get
bookkeeping overhead. If they knew that the patch was suitable for
stable couldn't they just cherry-pick them selfs. But that of course
would mean they have to at least compile the change and testing would
be good. So guess both add more work.

I think that its that what it all boils down to. How to get the most
out of a stable branch with the least amount of work.

When working on the svga driver for linux I had a set amount of
developer and QA time that I had at my disposal. We choice to build
the driver from the 7.7 branch, mostly because this was the branch
that the windows driver was based off so we knew that all general 3D
testing and bug fixing would apply to our release as well, but also
this happen to be the branch that happen to be around that time we
started doing stabilization work. Given these parameters for me the
logical thing was to have QA test 7.7 and report bug, then fix the
bugs in 7.7 and then "waste" as little time as possible getting these
fixes into master (that is merge and fixup any build breakage if
somebody complains).

Now maybe this is where our mindsets differ, maybe we care more about
what goes into a release/stable branch then what happens to go into a
future release/master branches. Maybe the "VMware" way is shortsighted
in that it only care about the stable branch, but for me it gives me
the most found and fixed bugs affecting the stable branch. Or maybe
the problem is that we have to long cycles (pretty much skipping every
other mesa release)?

If anybody have a better way to get more bugs found and fixed into a
stable branch then directing all available resources on that branch
then please let me know.

If I where to do bug-fixing on the master branch I run the risk of
having to fix the bug twice. Sure its good that the bug got fixed in
master as well but its time that gets taken out of other bug-fixing
for stable. Add to that I also have to spend time verifying it myself
on both branches and then direct QA to verify on their setup.

I guess what it does boil down to is where you do your main
development. For me, I did all the work on the stable branch and every
once in a while when I got time over I did work on master. Once we
stopped merging back 7.7 to master/7.8 keeping track of what has gone
from 7.7 to master/7.8 became pretty damn hard, well the first time
wasn't that hard just "git log --pretty=format:%(hash) master..7.7 --
touching/path | xargs -n1 git cherry-pick" tho following runs did not
go well since doing two git cherry-pick of the same commit would
explode. And any conflict would cause me to do the whole thing
manually anyways. The bottom line is that keeping track of what I have
ported over from stable to master because hard after a while since
cherry-picking makes it impossible for git (is there a way?) to tell
if a commit has already been picked.

I would love to have a way where I could minimize any overhead and
maximize bug fixing time for both preferred ways of working (master vs
stable). But I hope that you can understand that I feel it would be
wrong to make things harder for someone doing bug-fixing on a stable
branch then someone doing unstable work and one off bug-fixes on
master. Again not that this is unimportant. I have to thank a lot of
people for doing testing and trying patches for fixing regressions and
bugs on the master branch.

Tho I wonder if some sort of hybrid approach could be done, merging at
first and then switching over to cherry-picking between the branches
when they have diverged to much. Or maybe that just gives nobody what
they want.

Hmm thats a lot more then 2 cents but there you have it.

Cheers Jakob.