[Mesa-dev] [RFC] Mesa 17.3.x release problems and process improvements

[Mark Janes]

Daniel Vetter <daniel at ffwll.ch> writes:

> On Mon, Mar 12, 2018 at 11:54:45PM -0700, Kenneth Graunke wrote:
>> On Friday, March 9, 2018 12:12:28 PM PDT Mark Janes wrote:
>> [snip]
>> > I've been doing this for Intel.  Developers are on the hook to fix their
>> > bugs, but you can't make them do it.  They have many pressures on them,
>> > and a maintainer can't make the call as to whether a rendering bug is
>> > more important than day-1 vulkan conformance, for example.
>> > 
>> > We could heighten the transparency of what is blocking the build by
>> > publicizing the authors of bisected blocking bugs to Phoronix, which
>> > might get things moving.
>> 
>> I hope you're being sarcastic here, or else I'm misunderstanding your
>> proposal.  Public shaming of developers who create bugs has absolutely
>> no place in the Mesa community, IMHO.  It would foster the kind of toxic
>> community that none of us want to be a part of.
>> 
>> Sometimes, people who create bugs are the very people that work the
>> hardest, who the project may not even exist without.  Would you want
>> to chew out someone for creating a bug in a Vulkan driver when...if it
>> weren't for that person, you wouldn't have a Vulkan driver at all?  Or,
>> maybe they caused a couple bad bugs...but also fixed hundreds of them.
>> 
>> Other times, they're new contributors or volunteers who do this, not as
>> their day job.  Frankly, those people are under no obligation to help us
>> at all, so we need to thank them and appreciate the time and effort they
>> spend - and give them a hand fixing things when they're too busy, or
>> don't have the relevant hardware or skill to track down a regression.
>> 
>> It's easy to be pissed off when there are bugs, and things seem to not
>> be making progress, but let's try and keep things positive and work
>> together to make Mesa the best we can.
>
> I'd like to second this with my experience from the kernel community. The
> public shaming game for when you create a regression is very strong there,
> lead by Linus Torvalds. In my experience this directly causes:
>
> - Maintainers to hide bug reports and regressions reports at all costs,
>   because having Linus destroy you just aint never worth it. The meta game
>   becomes "avoid getting railed" instead of "deliver quality code", and
>   there's lots of ways to easily achieve the former that serious hurt the
>   latter.
>
> - Best practice (in my experience) is to not mention the dreaded
>   "REGRESSION" tag when you need another maintainer's help to fix a
>   regression, because it's too likely they'll just panic. That means they
>   start screaming at you to go away, or brain locks up and they can't
>   effectively help you track down the bug (seen both cases).
>
> - Creates a culture where talking about process/tooling improvements to
>   prevent regressions and/or handle them quicker becomes too dangerous,
>   because it all turns into a personal shaming game of who maintains the
>   worst subsystem.
>
> Long term you end up with a culture fucked up for good :-/
>
> Imo the only way to make this better is to try analyzing why a regressions
> happened, and fix the tooling to prevent that in the future. Maybe better
> test coverage (and long term efforts to fix known gaps), maybe better
> presentation of automated checks (stuff like github pull requests that
> automatically run CI and report full results, blocking the merge if
> anything is amiss).

You have to have a very strong CI to use it to block commits.  i965 Mesa
has a big CI which identifies many regressions, but I wouldn't want to
checkpoint commits in an automated way.  A large pool of obsolete
CI hardware will have lower reliability than the mesa master branch --
which generates noise for developers and impedes progress.

> Personally I have high hopes for gitlab.fd.o to enable us to do a lot of
> that automation in a much better and much more discoverable way, but it's
> some ways in the future still. Besides better quality that would also help
> us ramp up new contributors, since instead of unwritten rules they'd not
> just get documented merge criteria, but have a pile of bots that
> interactively walk them through everything (the best projects auto-insert
> a comment from the bot with instructions how to repro results if anything
> fails, with links to further docs).
>
> Assume that people try to do the best and fix the tooling/support
> infrastructure to allow them to, and they will deliver. Blaming them just
> drives them into hiding and looking for better places to have fun.
> -Daniel
> -- 
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch