[Mesa-dev] so the development model is working?

[Dave Airlie]

>
> I develop on multiple machines too, FWIW.  I do lots of ssh stuff like
> Keith described.
>
> BTW, your last sentence can go either way.  From my point of view
> it's: "Its a lot easier to just fix bugs on the 7.8 branch on those
> machines, and when you get around to it later merging the changes back
> into master, and dedicating a day to retesting master on as many
> configurations as you think require it for you to be happy with the
> work done."

The thing is breaking master by accident or temporarily isn't near as
bad as screwing up on the stable branch. I really want stable branch
to be something I can pull a diff from at any point without worrying
its broken stuff. I don't think we are good enough to stabilise a
single driver or piece of mesa on the stable branch without causing
collateral damage or unknown side effects in other drivers or parts of
the codebase. Bugs are a natural side effect of any development, and
if there are more people using master for development there is a
chance of someone else tripping over the bugs, before hitting stable.

>
>
>>> I'm usually working on master.  If I think I've found a bug (or
>>> someone reports a bug in 7.8, for example) I just change terminal tabs
>>> and start debugging in my 7.8 directory.  I don't have to save my work
>>> on master or check out a different branch, etc.  Just switch to a
>>> different terminal and fire up emacs.  After I fix the bug, I commit
>>> it to the 7.8 branch and document it in the release notes file (I'm
>>> practically the _only_ person who bothers to do that, btw!).  If I
>>> think it's important to get the fix into master quickly (or there's
>>> any doubt about applicability) I'll do a merge right away.  Otherwise,
>>> I just resume whatever I was doing before, knowing that the fix will
>>> propogate into master whenever we do a merge.
>>
>> Again I don't think is what the majority of developers at least the
>> part-time ones do or anything close. They restrict their limited time
>> to making master at any point in time work well for them. If you are
>> developing a feature, and you fix a bug in a separate commit while
>> working on the feature, taking a divergent path into 7.8 land is quite
>> a detour, it usually involved git rebasing the stable 7.8 you checked
>> out a month ago up to whatever is latest version, rebuilding it,
>> applying your patch, building, testing, pushing, pulling into master,
>> rebasing your current development work which may also need the bug
>> fixed to continue on top of it. I really don't see this as a small
>> divergence. If you are focused on doing development, this is a
>> complete task switch, which could reliably be done in a separate task
>> later, with arguably a lower chance of regressions.
>
> I think this is a matter of opinion because I think it's easy to
> temporarily switch to my 7.8 tree when I find a bug while working on
> master.  Also, if I'm working on file.c I don't have to temporarily
> back-out my new work there in order to fix/commit a bug elsewhere in
> the file.

git stash is what usually works here, or git add -p, I generally just
git add -p the fix, commit it to master, stash local changes, run
test, push commit to master, unstash and keep going. I often queue up
2-3 fixes in my master before I finish the feature or bother testing
all the fixes.

>>> I think you exagerate how many testing steps you have to do for each
>>> and every fix.  Absolutely, there are times when thorough testing is
>>> needed.  But lots of simple bug fixes/changes in 7.8 can be merged
>>> into master without worry.  I can provide many concrete examples if
>>> you don't believe me.
>>
>> I can also provide quite a large long list of commits that aren't in
>> 7.8 because nobody wants to deal with them. Also committing any fixes
>> directly to a stable branch is something I've never really heard of as
>> a standard. I would assume the expectation is that nobody willing
>> introduces regressions, but lots of people accidentally do, and it
>> would be better to find those in master first not in the stable branch
>> when a distro has already shipped it.
>
> If you're concerned with causing regressions with a commit, you can do
> several things including posting the patch for review first and/or
> running a bunch of regression tests (which it sounds like you're
> already doing) before committing.  Also, just because a bug fix works
> on master is not a guarantee that it won't cause regressions on the
> stable branch.  It goes both ways.

The problem with regression tests is they don't cover enough hw
combos. Like I can regression test some radeon easily, however when I
shove stuff onto master, I get the side benefits of Intel testing for
regressions in their driver, the other radeon guys doing tests on
different hw combos etc.

>> But they are stabilizing master for the next release. For r300 for
>> example accel CopyTexImage was added in master, while fixing that
>> Maciej also fixed a few FBO bugs. His focus was just on making master
>> pass piglit better. He wasn't thinking about how applicable these
>> fixes were to stable, he probably didn't consider the bugfixes to be
>> stable material. Once master diverges from stable by more than a
>> feature or two, most developers will need to work on master to make
>> sure the features introduced are better tested, you don't want someone
>> testing on 7.8 on the offchance that a feature in master will
>> invalidate the tests by having replaced a codepath.
>
> I'd argue that stabilizing the stable branches is more important than
> stabilizing master.  The release notes for a x.y.0 release clearly
> state that it's a new development release and people especially
> concerned with stability should stick with the previous x.(y-1).z
> release.
>
> Plus, before a new x.y.0 development release is made, Ian's been
> laying down a release timeframe that is meant for firming up the
> release.  That's the time to try to really stabilize master.

But I don't think that has happened at all, we've had a couple of
releases with this development cycle and from what I can see all the
developers are still developing on master.


>>
>>>                 Author             Signed- off-by
>>> Brian Paul         20                    18
>> Lots of gallium and mesa/st fixes - number of distros shipping
>> anything out of here? 0.
>>
>>> Jeremy Huddleston  18                    17
>> - apple builds fixes - 0 applicabilty
>>
>>> Ian Romanick        6                     1
>> - version numbers + GLX fix - the only real fix for insanity that
>> shouldn't have happened.
>>
>>> Dan Nicholson       4                     4
>> gallium + egl + build fixes - 0 applicablilty no distro ships a 7.8.x
>> gallium driver or EGL.
>>
>>> Vinson Lee          4                     0
>> - 2 useful builds fixes - one i915g fix (anyone shipping it?)
>>
>>> Jesse Barnes        4                     0
>> - some good DRI2 fixes, I'll allow Jesse to reveal his own opinions on
>> getting those in.
>>
>>> Matthiew Herbb      4                     0
>> gcc 3.3 build fixes - nice but nothing for a modern distro
>>
>>> Pierre Willenbrock  3                     0
>> - useful fixes
>>
>>> Alex Deucher        2                     0
>> - pci ids
>>
>>> Aaron Plattner      2                     0
>> - useful fix
>>
>>> Chia-I Wu           2                     0
>> - st/vega - does anyone ship it?
>>
>>> Maciej Cencora      2                     0
>>> Michel Danzer       2                     0
>>> Kristian Hogsberg   1                     0
>>> Ben Skeggs          1                     0
>>> Dave Arlie          0                     1
>>> (plus a few other one-timers)
>>
>> So my guess here is that the majority of these things done on the 7.8
>> branch have only a minor effect on the driver set that is currently
>> being shipped by distros and the bugs people are seeing in the field.
>> Do you at least agree with that?
>
> Yes, I do.  The point I was making is the development model I'm
> following works pretty well.  Regardless of the contents or audience
> of the 7.8.x branch, I feel each commit to the branch is a valid
> fix/improvement.  And it's not painful to work in that manner.
>
> If I were working on hardware driver (such as I did with i965 last year)
> I'd be following the same development model, making the i965 driver on
> the 7.8 branch better and better.
>
>
>
>> Any gallium or egl related fix isn't
>> useful to the consumers of this branch, other than I assume vmware
>> consuming it. I'm also assuming you use the stable branch for stuff,
>> again if I'm wrong please clarify.
>>
>>> If you're concerned about producing a stable driver, why aren't you
>>> making more fixes to the 7.8/stable branch, whether by cherry picking
>>> or whatever?  That's the whole point of it.  Master is not a stable
>>> branch.
>>
>> Is cherry-picking not frowned upon?
>
> It is, but it hasn't stopped people.  After merging, those cherry
> picks appear in the master history twice, unfortunately.  But I'd
> rather have that than no fixes to the 7.8 branch.  I'm more concerned
> with getting bug fixes into the stable branch than maintaining a
> pristine git history.
>
>
>
>> I thought the whole point of the
>> development model was that we don't cherry-pick stuff. If we are
>> allowed to have two different development models, where one group
>> cherry picks and one group pushes stuff to 7.8 and pulls, I think
>> we'll end up with a lot worse conflicts. Cherry-picking to 7.8 and
>> pulling back into master just seems to make a bad situation more
>> painful. If you are entirely focused on 7.8 how do you know nobody
>> hasn't already fixed something in master and just used the wrong
>> development model..
>
> Scan/read the git commit emails.
>
>
>> Hence why I think only the cherry-pick model
>> actually works, and the non-cherry-picking model has just become
>> something nobody is taking seriously at least with shipping code.
>
> Yeah, that's unfortunate.  If people would give it a try for a while
> they'd get used to it and see it's not as hard as it's been made out
> to be.  And again, periodically doing a merge is a lot quicker/easier
> than cherry-picking oodles of individual commits.  I've tried both
> ways - I know from experience.

>>> Look above and see if you can guess why I prefer doing merges to
>>> cherry-pick.?  I'd rather do 3 merges vs. 20+ cherry-picks.  Cherry
>>> picking quickly becomes a PITA once you get beyond a handful of
>>> patches or one commit per week or so.
>>>

Here's my other problem I didn't really clarify, from what I can see
your model just doesn't scale well. If there are 15 developers working
on this model, and only every 30-40 commits someone pulls 7.8 into
master, they are going to get a bunch of conflicts they've no idea wtf
to do with. So to make it useable for more than 1-2 developers, each
developer needs to pull 7.8 into master after each set of fixes they
do, if thats 1 fix or 50. Maybe if it doesn't conflict they can avoid
pushing master, but if there is any hope of a conflict with master,
they must push it. Now you end up with a merge:fix ratio of close to
1:1 and you really don't gain anything from this model. This works
fine now because nobody is doing it, but I can't see how you think it
will scale if everyone starts doing it, especially in areas where a
few people work. I'm interested if you think this analysis is correct?
maybe I'm missing something in how this scales. With your example
there you'd end up doing closer to 10-15 merges. Not to mention the
side effects Eric and Kristian pointed out with stable collateral
regressions in master due to mismerges or simply being able to fix
something nicer in master.


>>
>> The thing is I'm trying to take a step back from being a mesa
>> developer to my other task which is being a mesa consumer and from
>> what I can see I'm not getting the stuff to package that I get from
>> various other projects, i.e. the kernel + X server/drivers. Whether
>> that means we need a mesa stable maintainer and I should stand up and
>> take on that task and only one person should be in charge of the
>> stable branch might be something worth investigating.
>>
>> One idea I did have was that maybe people working on master could tag
>> commits they'd like to see considered for stable with a line in the
>> git commit, like the kernel has cc: stable at kernel.org. Then a stable
>> maintainers can try pulling those back into stable, announce when they
>> had something worth releasing, get everyone to start a test run on
>> their hw and then release a final tarball, every 2-3 months.
>
> Getting everyone to do anything is hard.  And I don't know who has the
> time to handle that role.  My work days at VMware are pretty full and
> I don't have as much spare time for Mesa work as I did in the past.

I'd be happy to try and take on some of this if people think its a
good plan, i.e. if I can persuade people to tag commits in master with
some common notation, it would be quite easy to automate
cherry-picking them to stable and dropping any that didn't auto-pick.
Its also quite hard when master and stable have diverged a bit in
terms of features to know if the features you fix bugs in, even exist
in master. I know at least for Maciej's r300 fixes a lot of the code
in the fbo/texture code was different so he probably didn't suspect
they code was in 7.8 in the first place. I know I've lost track when
master was 2-3 months past 7.7 of what was in 7.7 and what wasn't. As
gallium accelerates branch merges this isn't going to get easier.

Dave.