[Libreoffice-qa] Fwd: Re: Closing NEEDINFO bugs

[Bjoern Michaelsen]

Hi Alex,

On Sun, Aug 19, 2012 at 10:14:38PM +0200, Alex Thurgood wrote:
> If you look more closely, quite a few of these (I have no stats to
> back me up) were reports that Bjoern had reset in November to
> NEEDINFO when he did his first clean-up off the cuff, by resetting
> declared bugs to the NEEDINFO status. Since many of the OPs at the
> time were unaware what that meant, 

How so? There was a link accompanying that change asking to put the bug back
into status NEW and two links explaining the background.

> My own personal take on this is that if you guys want to regularly
> schmeiss out bug reports that someone else has happened to
> sweepingly set as NEEDINFO, then I'm wasting my time here in QA.

That these old bugs were moved to NEEDINFO was an one-off artifact of the bugs
having been implicitly confirmed in the old bugzilla, which started the bugs in
NEW, not UNCONFIRMED. 

> In my opinion, you have thrown the baby out with the bath water, but hey,
> just carry on like that and the world will be well. I know, I'll think I'll
> just wait 3 months and set a load of bugs to NEEDINFO, and then 3 months
> later strike them all out as INVALID - sounds fair ? No, didn't think so.

Actually, closing old, incomplete and inactive bugs is not at all uncommon.
Launchpad for example does that automatically: A bug that is marked incomplete
is closed as "expired" after 90 days.

> (a) ignore the disappointing approach from QA and leave LO to the
> hell it is getting itself into ;
> 
> (b) vociferously tell the LO project to p*ss off (we've had at least
> two or three of those already)
> 
> (c) maybe, just maybe, grit my teeth and reset the issue to
> re-opened, IF, and only IF, I give a damn.
> 
> The statistics you invoke as justification only take account of
> option (c), which is a false assumption of social behaviour.

The question is: Would we as a project better off, if we leave the bugs in
NEEDINFO? Please take a look at:

 http://skyfromme.wordpress.com/2012/08/16/on-the-importance-of-being-a-bug-confirmer/

and especially the table that is linked in there. The first job of QA -- as
hard as it sounds -- is not to handguide each and every bug reporter along. We
would love to do that, but given our limited resources and being purely
volunteerbased that is simply not the first task of the QA team. The first goal
is to find the most embarassing and urgent bugs and hand them over to the
developers to take care of them. If you look at the table linked in the above
blogpost:

 https://bugs.freedesktop.org/report.cgi?x_axis_field=resolution&y_axis_field=bug_status&z_axis_field=&query_format=report-table&short_desc_type=allwordssubstr&short_desc=&product=LibreOffice&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&bug_status=NEEDINFO&bug_status=PLEASETEST&longdesc_type=allwordssubstr&longdesc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_id=&bug_id_type=anyexact&emailtype1=substring&email1=&emailtype2=substring&email2=&emailtype3=substring&email3=&chfieldvalue=&chfieldfrom=&chfieldto=Now&field0-0-0=noop&type0-0-0=noop&value0-0-0=&format=table&action=wrap

You find there are over 5000 open bugs and >200 of those are assigned to
developers. The first goal of the QA team is to make sure that these bugs are
the most important to address. So QA has to ensure those bugs are of the right:

- quality (good triage and critical bug)
- quantity (Developers should always be aware of the most important bugs to work on)

Given that 87 of the assigned bugs have not seen a change in more than 180
days, it is clear that it would not help much to push more bugs in the
direction of development in its current state of operation. So what remains for
QA to do, is to find the most important bugs and from those the ones that are
best triaged: have a reproduction scenario, a pinned down version when the bug
appeared, a bibisect. Given the limited resources the team currently has and
5000 open bugs, we have to focus on those bugs -- and they are unlikely to be
found in those bugs in state NEEDINFO for 6 months. _If_ they are, they
hopefully will get reopened quickly.

> Sorry for what appears to be a rant, but this whole sordid affair
> has left me with a very bitter taste in my mouth, it was bad enough
> the first time around, and now this comes as the icing on the cake.

Yes, we should have learned from the first bulk change, and the execution on
this one was quite a bit unfortunate. The next time this should:

- be announced on _all_ the relevant list a 1-2 weeks before it happens
- on the planet with a blogpost
- best be two-staged:
  - first comment on the bug: If this bug does not receive the requested
    information (or confirmation by another contributor) it will get closed in 14 days
  - then close those that still are untouched 14 days later.

> Well, if that's how the project wants to play, so be it, but I'm a
> hair's breadth away from walking away from it. What has been played
> out here is clearly an attempt to alleviate a perceived lack of
> control of increasing bug count within the project. It might not
> have been planned that way, but that is how it looks to the outside
> and casual bug reporter. And then we have the gall to say that we
> need more people for QA - come on, who are we kidding if we act like
> that ?

We _need_ more people doing QA -- but we need them mostly for identifying the
most urgent and critical bugs out of the pool of all the open bugs. While
reporting new bugs is a valuable contribution too, for the project it pales
compared to those doing the tough job of bugwrangling, that is: finding
duplicates, confirming bugs, looking out for the most critical and urgent ones.

> My motivation for staying is directly linked to the usage I have on
> a professional level of the software with regard to databases. If
> mine, and others, bug reports can be swept under the carpet and then
> be told that all we have to do is reactivate them if we're not
> happy, well, I'd  be inclined to tell you all to take a running jump
> too. If I want hassle, I can go outside and pick a fight at the
> local pub, or for a quieter life, I can switch to competing software
> not a million miles away.

I totally understand the sentiment, but I have seen at Sun how dangerous it is
to raise expectations that are not fullfilled. There some developers had a huge
backlog of bugs assigned to them and always new ones flowing in. Because of
this, those bugs that the developer priotized lower in the set essentially
never got a chance to be considered as incoming, more urgent bugs bypassed
them. With every release there was an outburst of disappointment as specific
bugs were "still not fixed". I think we should avoid such a situation of
creating false hopes.

LibreOffice is still doing quite well on one metric: Bugs fixed
per unit of time, esp. compared to history and given the leaner team we are
working with. Bugwrangling (that is: finding duplicates, confirming bugs,
checking bug states and versions, identifying the most urgent ones) has to make
sure, we are fixing the _right_ bugs. Only after that comes bug reporting, that
gives us the bugs to select from.

There is a lot more honesty in this compared to 'that competing software not a
million miles away' -- and it also makes things a bit more grim. Still I think
it is the right thing to keep being honest. However, now that the project is
on some stable footing, we need to make sure we improve on what we have in many
areas:

- More early testing (esp. daily master build) to find urgent bugs early(*)
- Better bugwrangling (that need more people)
- Better communication between users and QA / QA and development
- Quicker bugfixing for urgent bugs (that cannot be solved in QA -- development
  need to adapt for that)

Best,

Bjoern

(*) Incedentally that helps to get bugs fixed too: When you can say: This broke
between day X and Y (either by running dailies or by using bibisect, it is much
more easy to pin this to the developer 'guilty' of breaking it and thus getting
him to fix it.