[Mesa-dev] [RFC] Mesa 17.3.x release problems and process improvements

[Daniel Vetter]

On Mon, Mar 12, 2018 at 11:54:45PM -0700, Kenneth Graunke wrote:
> On Friday, March 9, 2018 12:12:28 PM PDT Mark Janes wrote:
> [snip]
> > I've been doing this for Intel.  Developers are on the hook to fix their
> > bugs, but you can't make them do it.  They have many pressures on them,
> > and a maintainer can't make the call as to whether a rendering bug is
> > more important than day-1 vulkan conformance, for example.
> > 
> > We could heighten the transparency of what is blocking the build by
> > publicizing the authors of bisected blocking bugs to Phoronix, which
> > might get things moving.
> 
> I hope you're being sarcastic here, or else I'm misunderstanding your
> proposal.  Public shaming of developers who create bugs has absolutely
> no place in the Mesa community, IMHO.  It would foster the kind of toxic
> community that none of us want to be a part of.
> 
> Sometimes, people who create bugs are the very people that work the
> hardest, who the project may not even exist without.  Would you want
> to chew out someone for creating a bug in a Vulkan driver when...if it
> weren't for that person, you wouldn't have a Vulkan driver at all?  Or,
> maybe they caused a couple bad bugs...but also fixed hundreds of them.
> 
> Other times, they're new contributors or volunteers who do this, not as
> their day job.  Frankly, those people are under no obligation to help us
> at all, so we need to thank them and appreciate the time and effort they
> spend - and give them a hand fixing things when they're too busy, or
> don't have the relevant hardware or skill to track down a regression.
> 
> It's easy to be pissed off when there are bugs, and things seem to not
> be making progress, but let's try and keep things positive and work
> together to make Mesa the best we can.

I'd like to second this with my experience from the kernel community. The
public shaming game for when you create a regression is very strong there,
lead by Linus Torvalds. In my experience this directly causes:

- Maintainers to hide bug reports and regressions reports at all costs,
  because having Linus destroy you just aint never worth it. The meta game
  becomes "avoid getting railed" instead of "deliver quality code", and
  there's lots of ways to easily achieve the former that serious hurt the
  latter.

- Best practice (in my experience) is to not mention the dreaded
  "REGRESSION" tag when you need another maintainer's help to fix a
  regression, because it's too likely they'll just panic. That means they
  start screaming at you to go away, or brain locks up and they can't
  effectively help you track down the bug (seen both cases).

- Creates a culture where talking about process/tooling improvements to
  prevent regressions and/or handle them quicker becomes too dangerous,
  because it all turns into a personal shaming game of who maintains the
  worst subsystem.

Long term you end up with a culture fucked up for good :-/

Imo the only way to make this better is to try analyzing why a regressions
happened, and fix the tooling to prevent that in the future. Maybe better
test coverage (and long term efforts to fix known gaps), maybe better
presentation of automated checks (stuff like github pull requests that
automatically run CI and report full results, blocking the merge if
anything is amiss).

Personally I have high hopes for gitlab.fd.o to enable us to do a lot of
that automation in a much better and much more discoverable way, but it's
some ways in the future still. Besides better quality that would also help
us ramp up new contributors, since instead of unwritten rules they'd not
just get documented merge criteria, but have a pile of bots that
interactively walk them through everything (the best projects auto-insert
a comment from the bot with instructions how to repro results if anything
fails, with links to further docs).

Assume that people try to do the best and fix the tooling/support
infrastructure to allow them to, and they will deliver. Blaming them just
drives them into hiding and looking for better places to have fun.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch