RFC: group maintainership for misc drm trees

Wed Feb 17 13:40:09 UTC 2016

On Tue, Feb 16, 2016 at 10:44 PM, Daniel Vetter <daniel at ffwll.ch> wrote:
> On Tue, Feb 16, 2016 at 11:44:52AM +1000, Dave Airlie wrote:
>> On 15 February 2016 at 20:06, Daniel Vetter <daniel.vetter at ffwll.ch> wrote:
>> > Hi all,
>> >
>> > I've already chatted with some of you in private, here's the entire idea with a
>> > bit more thought. My motiviation for group maintainership of drm-misc was that I
>> > got a bit a guilty feeling the last few vacations/conferences when folks pinged
>> > about reviewed/tested pretty patches not landing. But also just increasing the
>> > bus factor and sharing the load better is good. And finally a shared misc drm
>> > tree would allow fringe drivers to get faster into Dave's drm-next by
>> > piggy-packing on top of the one pull request train. And it would also reduce a
>> > bit tree proliferation (at one point we had drm-misc/-bridge/-panel/-trivial and
>> > may more even). And at least everyone I chatted with seems to like the idea in
>> > principle.
>> >
>> > But what's still open is how to do it exactly. One big change with group
>> > maintainership is that you can't rebase a tree anymore. And right now I need to
>> > rebase drm-misc fairly often to throw out bad apples again. I think solving that
>> > is the important bit to make a shared drm-misc work. A few ideas:
>>
>> This is kinda what I don't like. I don't want a tree that can't rebase
>> out bad stuff
>> coming to me that often. If we are being too over eager in merging
>> stuff the answer
>> is to merge less not try and merge more. If we have a tree where things
>> are thrown until they stick I'm not sure the history will ever be nice
>> to pull from.
>>
>> I'd rather consider a staging tree where everything from patchwork can
>> get thrown
>> into, CIed, but that we cherry-pick fixes things to go back to me from the what
>> works category.
>
> I fully agree on your concern that drm-misc could become a mess - that's
> why I think everything bigger than e.g. 2 patches or touching drivers
> needs to soak first in a topic branch. We could still get it into drm-misc
> with cherry-picks to avoid too many merges.
>
> The other part where I often had to squash in fixups is build fail on arm.
> But I fixed myself with some decent cross-compile toolchain, so I'm
> positive that won't happen any more.
>
> I think a good idea would be that I test-drive this process a bit without
> adding more maintainers, i.e.
> - no more rebases for drm-misc
> - topic branches for everything big
> - dutifully testing arm before pushing
>
> I'll do that over the next few weeks, then we can see how bad it would
> look and whether we need more to avoid bad history.
>
>> I'm finding with i915 for example there is a massive latency in the pipeline now
>> waiting for fixes, and the pipeline to the end of drm-intel-next is
>> very long and
>> hard to figure out what fixes should be pulled back, and how much of the
>> driver has been rewritten between the -next and the -fixes pulls.
>>
>> > - I think CI is super-important. We're starting to finally roll that out for
>> >   real for i915, and it's catching an awful lot of stuff already. Not yet ready
>> >   for prime-time on public mailing lists, and for misc we probably can't test
>> >   every patch before they land like we do for i915. But CI should have veto
>> >   power before a pull request goes to Dave imo.
>> >
>> >   For non-i915 Daniel Stone and others are working on ARM CI using the generic
>> >   igt testcases for kms. And I'm open to merging driver-specific tests into igt
>> >   too, if it makes sense, and e.g. Eric has already pushed some vc4 tests.
>> >
>> > - Stuff needs to at least compile cleanly before pushing. I've been really bad
>> >   at that wrt arm drivers with my own drm-misc, but turns out it's fairly easy
>> >   to get this right: http://blog.ffwll.ch/2016/02/arm-kernel-cross-compiling.html
>> >
>> > - Bad apples need to be kicked out with reverts, not rebases. I think that's
>> >   fine for simple patches, and hence those can go directly to drm-misc. But a
>> >   bunch of subsystem-wide refactorings go in through mis trees, and for those
>> >   constantly mass-reverting until it's all solid is silly. And ime you need some
>> >   soak time in a shared tree to iron out bugs with those kind of endeavours. We
>> >   can address that with ad-hoc&short-lived topic branches which are then again
>> >   owned by a single maintainer, but automatically pulled into an integration
>> >   tree. After some soaking time to give CI systems time to crunch through those
>> >   topic branches can then be merged into the main drm-misc and removed.
>>
>> I think integration trees are probably the way forward, but I
>> understand they come
>> with a massive overhead for constant merging, I like the idea of topic
>> branches until
>> it becomes my turn to spaghetti merge 3-4 features at once. At that
>> point I still
>> fallback to the slow things down, pick one feature merge it, back the
>> others off,
>> and I think this is something we should do more off, the pile everything in and
>> hope that magical CI will stabilise things doesn't seem to be a responsible
>> process to me.
>
> My idea is that you won't see the integration tree mess ever, but that
> tested stuff would land in drm-misc first to resolve conflicts. But really
> there never should be, since when a series takes longer than 1-2 weeks to
> get ready it's just not ready for merging. And the entire topic branch
> should be kicked out.
>
> Also integration tree mangling is why I suggested we start out using dim
> scripts - we're integrating 5-10 branches for drm-intel-nightly since
> years, since a few months with multiple maintainers pushing conflict
> resolutions, and it all works really well.
>
> Also agreed that CI won't magically make stuff stabilise - if CI
> complaints the rule needs to be to revert or drop the topic branch, not
> try to wrestle it until it's less unhappy. That indeed just doesn't work,
> and we're learning all that with drm-intel right now (before we had no CI
> but ran on hope, which works even less).
>
>> so I'd like CI to be happening but I'd like it to be happening before the git
>> history is baked into stone.
>
> Yeah that's what we do for drm-intel. At least right now we don't even
> have anyone doing arm CI, much less CI before merging. Daniel Stone is
> working on it though, so I'm hopeful. But yeah for my own stuff or patches
> from intel folks we already throw them all at CI before they land
> anywhere.
>
>> > - Also we can't roll forward to latest drm-next easily with rebases any more. So
>> >   that needs to happen either right after the pull request lands (when no patch
>> >   has been merged meanwhile). Or with backmerges, which then need a short commit
>> >   message as to way the backmerge is needed ("Backmerge because we need
>> >   $feature" is what I usually type), plus sob line from the committer. And of
>> >   course the backmerged treed must be stable/non-rebasing and preferrably the
>> >   backmerge should be a release/pull-request tag.
>> >
>> > - For tooling I suggest we just go with drm-intel maintainer-tools for a start.
>> >   Picking dim has a few reasons:
>> >   * Well tested, documented and fairly complete (includes e.g. bash-completion).
>> >   * Integrated support for topic branches, including support for git worktree.
>> >   * Well-exercised and robust integration tree support.
>> >   Short-term we could add some convenience functions (e.g. for creating/merging
>> >   drm-misc topic branches). Long-term we might or might not want to have this
>> >   separated from drm-intel - that should be possible but a bit of work. Also,
>> >   this way drm-misc would stay at it's current location while we figure things
>> >   out. Longer term we might also want to look into adding other big drivers
>> >   into an over drm integration.
>> >
>> > Of course group maintainership needs an initial group. My experience from
>> > drm-intel is that a bigger group of maintainer has benefits: It's clear that
>> > part-time maintaining is ok too, with maintainers focusing on their area of
>> > interest/expertise and only helping out in other places when there's a gap (due
>> > to e.g. vacations). Anyway, here's my thoughts for the starting group:
>> > Archit, Jani, Thierry & me as existing maintainers of drm-misc/bridge/panel,
>> > Alex&Rob as maintainers of big drivers and engaged in core drm stuff,
>> > Daniel Stone so that he has no more excuses to stall on arm drm ci.
>> >
>> > I think if this goes well we can extend it to more driver maintainers, e.g.
>> > Patrick would like to just push gma500 patches to some tree and not fiddle with
>> > pull requests all the time himself.
>>
>> Is pull request sending a big issue? I don't really want a misc tree
>> as a way to avoid sending distinct pull requests for drivers. If we do
>> this I want
>> it to be drm core patches and subsequent fallout only and if we want
>> an unmaintained
>> drivers patches tree to do that separately.
>
> Eric and Patrick mentioned in private that pull reqs for drivers is
> somewhat a nuisance for them. And iirc Thierry also said he's drowning, so
> often only gets around to stuff very late.

I don't mind sending pull requests. It just felt overkill to do a PR
for a few trivial fixes. If keeping gma500 stuff separated from
drm-misc is easier for you and Dave I don't mind doing that.

-Patrik

>
> But really driver trees for small drivers is an entire different topic
> that I'd like to maybe keep in mind. But certainly not for initial.
>
>> I don't really like the misc tree replacing panel/bridge tree either,
>> those were nicely
>> compartmentalised in my head as trees I pulled but didn't really care
>> about as Thierry
>> did a good job and I don't have ARM hw that I personally care about.
>
> Hm, I mentioned that since I've heard complaints from ARM folks about
> tree proliferation, so figured we could try something to get things
> together a bit more. But since I care about x86 first I don't mind at all
> if we keep drm-misc restricted to what I pulled in thus far if you don't
> want this.
>
>> It's important to me that stuff still comes in some sort of
>> compartments, as it decides
>> for each pull request how much time/effort I put into testing it on my
>> end, I'm pretty much
>> at the mercy of ARM hw maintainers, but with x86 I still like to smoke
>> test it on some
>> hardware as much as possible, also with DRM core stuff.
>
> If we keep bridge/panel out I don't expect stuff will change at all in
> that regard for you. drm-misc will still be misc random patches, plus the
> oddball treewide refactor. And I'll still try to pace the bigger refactors
> reasonable. That needs some coordination among maintainers, but I think
> the bigger upside here is to get the oddball ones in with less overhead -
> big patch series are much harder to forget about ;-)
>
>> So I'm still not sure where I stand on this, personally I don't care
>> if you take holidays
>> and stuff stalls, delays aren't the big enemy here, merging crap is
>> much worse than
>> delaying.
>
> One part of my motivation is that this went much, much better than I ever
> hoped for in drm-intel. There's teething issues for sure, but it helped
> massively in increasing the bus factor, and in reducing people pinging me
> directly for every little tiny patch. So much more time for me to look at
> patches that are actually important for me to look at wrt "should this go
> in or not".
>
> Anyway no rush at all here for drm-misc, and I think next step is for me
> to test-run the "no rebasing" to figure that part out. But still would
> like comments/ideas from others, too.
>
> Cheers, Daniel
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch
> _______________________________________________
> dri-devel mailing list
> dri-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel