RFC: group maintainership for misc drm trees

[Thierry Reding]

On Tue, Feb 16, 2016 at 11:44:52AM +1000, Dave Airlie wrote:
> On 15 February 2016 at 20:06, Daniel Vetter <daniel.vetter at ffwll.ch> wrote:
> > Hi all,
> >
> > I've already chatted with some of you in private, here's the entire idea with a
> > bit more thought. My motiviation for group maintainership of drm-misc was that I
> > got a bit a guilty feeling the last few vacations/conferences when folks pinged
> > about reviewed/tested pretty patches not landing. But also just increasing the
> > bus factor and sharing the load better is good. And finally a shared misc drm
> > tree would allow fringe drivers to get faster into Dave's drm-next by
> > piggy-packing on top of the one pull request train. And it would also reduce a
> > bit tree proliferation (at one point we had drm-misc/-bridge/-panel/-trivial and
> > may more even). And at least everyone I chatted with seems to like the idea in
> > principle.
> >
> > But what's still open is how to do it exactly. One big change with group
> > maintainership is that you can't rebase a tree anymore. And right now I need to
> > rebase drm-misc fairly often to throw out bad apples again. I think solving that
> > is the important bit to make a shared drm-misc work. A few ideas:
> 
> This is kinda what I don't like. I don't want a tree that can't rebase
> out bad stuff
> coming to me that often. If we are being too over eager in merging
> stuff the answer
> is to merge less not try and merge more. If we have a tree where things
> are thrown until they stick I'm not sure the history will ever be nice
> to pull from.
> 
> I'd rather consider a staging tree where everything from patchwork can
> get thrown
> into, CIed, but that we cherry-pick fixes things to go back to me from the what
> works category.

This is kind of what drm-trivial was meant to be (at least for drivers
not necessarily core stuff), although I didn't see much benefit in (and
hence gave up on) maintaining such a tree if at the end of the day you
went and picked up the patches manually anyway.

My main pet peeve at the time had been that whenever you make changes
across all of DRM we're not very good at picking things up. Technically
maintainers would need to ACK such patches and then we'd have to wait
on either you or Daniel to pick them up.

I think it'd be really great if we had a tree where we can make these
kinds of patches available earlier. My experience has been that most of
the trivial errors are detected once patches hit linux-next. The
alternative to drm-trivial is to establish -next trees for all drivers.
At the moment there's only a handful of those, and not even radeon/amd
or nouveau feed into linux-next. With all the CI infrastructure that's
been created over the last few years I think we're missing out on much
of this goodness by not getting more code into linux-next early on. And
there's also additional testers which use linux-next in day to day use
and will report back when things break (beyond build issues or obvious
failures such as those that would panic unconditionally on boot).

Of course for separate driver trees to work well they do need to be in
a well-maintained state, and I'll be the first to admit that that's not
at all as easy as it may sound.

Feeding code into linux-next comes with an increase in responsibility as
well, because if we merge patches that break builds we make life
miserable for everyone else. Of course waiting until the merge window
before merging code is not a good solution either because you'll simply
postpone the misery. Probably the simplest answer here is to have some
standard sanity build scripts that will give us some confidence in that
what we merge at least builds fine. I've been maintaining a set of
scripts to do exactly that, and I'm hoping to find the time to polish
them a little more and perhaps write something up so that more people
can use them.

> I'm finding with i915 for example there is a massive latency in the pipeline now
> waiting for fixes, and the pipeline to the end of drm-intel-next is
> very long and
> hard to figure out what fixes should be pulled back, and how much of the
> driver has been rewritten between the -next and the -fixes pulls.
> 
> > - I think CI is super-important. We're starting to finally roll that out for
> >   real for i915, and it's catching an awful lot of stuff already. Not yet ready
> >   for prime-time on public mailing lists, and for misc we probably can't test
> >   every patch before they land like we do for i915. But CI should have veto
> >   power before a pull request goes to Dave imo.
> >
> >   For non-i915 Daniel Stone and others are working on ARM CI using the generic
> >   igt testcases for kms. And I'm open to merging driver-specific tests into igt
> >   too, if it makes sense, and e.g. Eric has already pushed some vc4 tests.
> >
> > - Stuff needs to at least compile cleanly before pushing. I've been really bad
> >   at that wrt arm drivers with my own drm-misc, but turns out it's fairly easy
> >   to get this right: http://blog.ffwll.ch/2016/02/arm-kernel-cross-compiling.html
> >
> > - Bad apples need to be kicked out with reverts, not rebases. I think that's
> >   fine for simple patches, and hence those can go directly to drm-misc. But a
> >   bunch of subsystem-wide refactorings go in through mis trees, and for those
> >   constantly mass-reverting until it's all solid is silly. And ime you need some
> >   soak time in a shared tree to iron out bugs with those kind of endeavours. We
> >   can address that with ad-hoc&short-lived topic branches which are then again
> >   owned by a single maintainer, but automatically pulled into an integration
> >   tree. After some soaking time to give CI systems time to crunch through those
> >   topic branches can then be merged into the main drm-misc and removed.
> 
> I think integration trees are probably the way forward, but I
> understand they come
> with a massive overhead for constant merging, I like the idea of topic
> branches until
> it becomes my turn to spaghetti merge 3-4 features at once. At that
> point I still
> fallback to the slow things down, pick one feature merge it, back the
> others off,
> and I think this is something we should do more off, the pile everything in and
> hope that magical CI will stabilise things doesn't seem to be a responsible
> process to me.
> 
> so I'd like CI to be happening but I'd like it to be happening before the git
> history is baked into stone.

I think topic branches work very well for this case. For Tegra I've been
maintaining an assortment of topic branches in a development tree that I
rebase onto linux-next (almost) on a daily basis. With the right scripts
this is fairly easy to do. It has the big advantage that you can keep
topic branches separate and merge them into one tree to test out the
result. Once the feature is ready you can "promote" it to one of the
stable branches.

I think for DRM this could work very well. There could be a single tree
for integration where almost everything gets pulled in. There could be
separate branches for the drivers, plus topic branches that track the
development of new features. Once a feature is ready, or maintainers
think that the driver code is done, the branches can either be cherry-
picked or merged into some stable branch that is the base for all other
development branches. Once patches are promoted to the stable branch(es)
the whole tree can be rebased and the promoted patches will simply be
rebased out of the respective branches.

This is all quite a bit of maintenance work, but I think with the right
tooling it could be made fairly painless. Group maintainership will also
help spread the load.

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/dri-devel/attachments/20160217/cdefe6b1/attachment.sig>