Bug Triaging FlowChart Example

[Joel Madero]

Hi All

I will be putting the latest workflow bug triaging flowchart on a wiki page
this afternoon with a link on the QA wiki page to it.  I saw an email in
the digest that had a ton of questions. I answered some to the best of my
abilities, if someone else wants to look at the questions (bolded) and
comment, please do so. I feel like as the new guy I really don't want to
overstep, mis-state, etc....

Here are my responses to the questions:

Just want to clarify a bit before addressing eah question. This is only
meant to be a guide for users, one example of many that is feasible to use.
Many people have said that they like the general guidelines but as was
mentioned earlier, there is no expectation what so ever that everyone will
follow this identically or at all really, it's up to the QA person if they
feel this is appropriate or not.

*I'd change the workflow a little bit by putting the obvious things at the
top:
- feature requests aka wishlist - add target milestone if a feature is
planned already
- bugs reported against not supported branches - exclude those at reporting
level by disabling old versions in select fields; but what to do when they
appear anyway -  mark them as INVALID at triaging? ask the reporter to check
in new version? Any comments?
- is it Most Frequently Reported - then just dupe it
and now, if a bug is still there, triage it by the proposed workflow.*
*
*
*
*
As of now there isn't the extra stuff, it may be included later, this all
began just a few days ago and I'm still trying to figure out how much to
include. Because this is only a general guideline, putting too much time
into detailing every step just makes it mechanical which this project will
never be. As for INVALID and other status', I've been asked to make another
flowchart for this and I intend on doing that soon (currently working on a
couple bugs so it's going to have to wait a bit)


*
- how you define most and many - I reported those two bugs on behalf of ~xx
people - is my bug better than other reports? Should this be controlled by
number of dupes? At what levels? Maybe enabling voting feature of Bugzilla
would help here...
*
Up to the person triaging and if I try to force my views, I'll get push
back ;) My general thought: most is >75%, many is >30%


*- regressions - this is a major problem and I think it deserves its own
place in the workflow. It is not a big problem for home users to switch the
version back and forth. But when we are talking in terms of bigger
deployment, when you have xx computers (and growing) it becomes a real
blocker sometimes. What good are new and fixed versions for, where there is
a regression which disqualifies the software for daily use? That is a big
problem.*

New one addresses this as a special note on the bottom. I agree that it's
important


*- backtraces - not all people can do proper backtrace, especially on
Windows, so maybe while triaging crash bugs one could ask for one a QA
member? BTW: Having own dev build of LO 3.5.4.2 I can be QA contact for
missing bt for all Windows crash bugs (except Base) on that branch. *
*
*
If you can't do a backtrace on something that needs a backtrace, the bug
shouldn't be triaged by you IMO. Many bugs don't require backtracing, those
that do should be left to people who know how to do it to triage.My
backtracing abilities are nill but on the list of to learn so until I
learn, I skip over the questions that need a backtrace.


*- permissions - seems like every user has canconfirm (Can confirm a bug)
and
editbugs (Can edit all aspects of any bug). Did you think about changing
that, so one could not interfere with QA actions or vandalize the bug
reports? *
*
*
I addressed something similar and ultimately we want to keep it open. There
is no vandalizing bugs, sometimes mis-triaging occurs (many times) but
hopefully with our team of qualified QA people most bugs are done
correctly. When someone reports it's left as UNCONFIRMED and from what I
see, only people from our staff really change that status. Until it's out
of that status, we know that most likely what was reported was by the user
and not our staff (ie. we check things)


*Bugs triaged.  But those are only b.f.o. bugs. Checking release info for
3.5.5.1 I can see such acronyms like bnc, i, rhbz and there are bugs
reported on b.f.o. coming from aoo (where they may be fixed already). Not
mentioning ubuntu, gentoo, opensuse and other bugtrackers. This is a very
serious problem, because not all bugs are in one place. Any comments about
that?*

I'm new to the team so I don't have a comment about this, maybe someone
else does though.


*So, not bad but if there is more not assigned RESOLVED FIXED or NEW bugs
with commits in the repo, then I see a problem here. QA should know what is
happening with the bugs without the need to follow the comments. How does it
look for xxxx commits? Also I see a lot of commits without bug report number
in it. What is fixed then? I see it as another problem.
*
Same response as above. As I get used to the workflow and the realities of
doing such a large project with such a limited number of users I'll see
what my opinions are. I made this to just clarify for myself and share with
others who might find it useful. This is out of the scope of the flowchart
but others may have opinions about it.


*I see you do not use QA field in Bugzilla. Why is that? Every component of
a*
*product can have its default QA specialist. Then developers responsible for
*
*a component quality can get a notice of new bugs instead of news about all*
*of them. Also QA specialist assigned here could be responsible for*
*bibisecting, bt delivery or verifying the bug.*

Not sure. I don't see it as needed really and I think with the limited
time, people and man hours (a bit redundant?), adding even more to fill out
is problematic.


*By the way - Browsing through dev/qa mailinglist I can see that you prefer*
*those lists than bugs in Bugzilla. For instance - are those action items*
*from ESC or QA meetings reported as bugs in Bugzilla? b.f.o is not used*
*project wide or am I missing something?*
*
*
Not sure that I understand, maybe someone else can answer it.

*Do you VERIFY bugs? I see just 62 bugs as VERIFIED FIXED. I do not see such
*
*section in Getting Involved. Surely there are users willing to participate*
*this way.*
*
*
Agreed, I'll put this on my to do list.

*I see you do not use flags and prefer Whiteboard. Why is that? Maybe a*
*better idea than a MAB nightmare (IMHO meta bugs with >300 comments are*
*useless) would be a release tracking flag, where QA would like to see a
fix.*
*Mozilla projects use them a lot for instance... *

I think too much is already written about using whiteboard correctly and
too many on the QA staff (as well as others) are comfortable with the
whiteboard. I am yet to use it really but don't see this as a major
obstacle personally.


I hope that I answered things correctly and others may have more feedback.
Appreciate the feedback. My goal is by end of year to be at 0 UNCONFIRMED
bugs > 2 weeks old. It's a hefty goal but I think with a workflow



Hi.
This is a very nice workflow, but I have some questions:
- how you define "Bug prevent users from making professional quality work?"
Interoperability issues are a big obstacle. If this package is supposed to
read/write MSOffice files, then every new bug in this area should have high
priority by default. I have reported two annoying:
https://bugs.freedesktop.org/show_bug.cgi?id=47138 47138  and
https://bugs.freedesktop.org/show_bug.cgi?id=49102 49102 . Are those
"prevent users from making professional quality work" enough? Home users can
live with a workaround, but it is impossible to expect that xx(x) users will
do it xx times a day. Everyday. That is not "professional" (even worse - it
is very "unprofessional" for those users migrated from MSO, where it just
worked).

I'd change the workflow a little bit by putting the obvious things at the
top:
- feature requests aka wishlist - add target milestone if a feature is
planned already
- bugs reported against not supported branches - exclude those at reporting
level by disabling old versions in select fields; but what to do when they
appear anyway -  mark them as INVALID at triaging? ask the reporter to check
in new version? Any comments?
- is it Most Frequently Reported - then just dupe it
and now, if a bug is still there, triage it by the proposed workflow.

- how you define most and many - I reported those two bugs on behalf of ~xx
people - is my bug better than other reports? Should this be controlled by
number of dupes? At what levels? Maybe enabling voting feature of Bugzilla
would help here...

- regressions - this is a major problem and I think it deserves its own
place in the workflow. It is not a big problem for home users to switch the
version back and forth. But when we are talking in terms of bigger
deployment, when you have xx computers (and growing) it becomes a real
blocker sometimes. What good are new and fixed versions for, where there is
a regression which disqualifies the software for daily use? That is a big
problem.

- backtraces - not all people can do proper backtrace, especially on
Windows, so maybe while triaging crash bugs one could ask for one a QA
member? BTW: Having own dev build of LO 3.5.4.2 I can be QA contact for
missing bt for all Windows crash bugs (except Base) on that branch.

- permissions - seems like every user has canconfirm (Can confirm a bug) and
editbugs (Can edit all aspects of any bug). Did you think about changing
that, so one could not interfere with QA actions or vandalize the bug
reports?

Bugs triaged.  But those are only b.f.o. bugs. Checking release info for
3.5.5.1 I can see such acronyms like bnc, i, rhbz and there are bugs
reported on b.f.o. coming from aoo (where they may be fixed already). Not
mentioning ubuntu, gentoo, opensuse and other bugtrackers. This is a very
serious problem, because not all bugs are in one place. Any comments about
that?

Does the developers have their workflow in Bugzilla? I compared few commits
(libreoffice/core libreoffice-3-5-5) with b.f.o reports:
- 50603 - not assigned, UNCONFIRMED>RESOLVED FIXED
- 43967 - assigned, UNCONFIRMED>NEW>RESOLVED
FIXED>REOPENED>ASSIGNED>RESOLVED FIXED
- 48601 - not assigned, UNCONFIRMED>NEW
- 50988 - assigned, UNCONFIRMED>ASSIGNED>RESOLVED FIXED
- 43249 - assigned, UNCONFIRMED>NEW>RESOLVED FIXED

So, not bad but if there is more not assigned RESOLVED FIXED or NEW bugs
with commits in the repo, then I see a problem here. QA should know what is
happening with the bugs without the need to follow the comments. How does it
look for xxxx commits? Also I see a lot of commits without bug report number
in it. What is fixed then? I see it as another problem.

I see you do not use QA field in Bugzilla. Why is that? Every component of a
product can have its default QA specialist. Then developers responsible for
a component quality can get a notice of new bugs instead of news about all
of them. Also QA specialist assigned here could be responsible for
bibisecting, bt delivery or verifying the bug.

By the way - Browsing through dev/qa mailinglist I can see that you prefer
those lists than bugs in Bugzilla. For instance - are those action items
from ESC or QA meetings reported as bugs in Bugzilla? b.f.o is not used
project wide or am I missing something?

Do you VERIFY bugs? I see just 62 bugs as VERIFIED FIXED. I do not see such
section in Getting Involved. Surely there are users willing to participate
this way.

I see you do not use flags and prefer Whiteboard. Why is that? Maybe a
better idea than a MAB nightmare (IMHO meta bugs with >300 comments are
useless) would be a release tracking flag, where QA would like to see a fix.
Mozilla projects use them a lot for instance...

Well, sorry for such a pack of off-topic questions, but I'd like to
understand QA in this project better.
Best regards.

On Thu, Jun 14, 2012 at 10:15 AM, Joel Madero <jmadero.dev at gmail.com> wrote:

> Just an update on this since some people showed interest. I am/was
> willing to do a manual list using a quick macro in calc but the wiki
> blocks me from using href so unless that is changed I don't see how I
> can link the bug to FDO which makes this endeavor a bit useless. If
> someone knows if I can get around the spam block I'll get the page
> updated, otherwise going to say this one is off :(
>
>
> Joel
>
> On Sat, Jun 2, 2012 at 3:39 PM, Joel Madero <jmadero.dev at gmail.com> wrote:
> > I've never used this kind of API before. I could just use it within the
> wiki
> > at Libre Office Development? I apologize for the lack of knowledge,
> trying
> > to pick it up and contribute just need a little guidance. Is there a how
> to
> > somewhere that I can look over?
> >
> > I'm going to say I'm a beginner but not absolute beginner so that gives a
> > bit of a scope of my abilities. Thanks again to all of you
> >
> > On Sat, Jun 2, 2012 at 3:03 PM, Markus Mohrhard
> > <markus.mohrhard at googlemail.com> wrote:
> >>
> >> Hello Joel,
> >>
> >> 2012/6/2 Joel Madero <jmadero.dev at gmail.com>:
> >> > In order to use the API would one of the webmasters have to add things
> >> > to
> >> > the server? Looks much more promising than it did a few hours ago.
> >> > Thanks
> >> > for pointing me in the right directino
> >> >
> >> >
> >>
> >> I did not have success using the REST Api with the fdo bugzilla. As
> >> much as I understood from the documentation it is an optional module
> >> and it seems it is not activated at fdo. However there is another
> >> package that we now succesfully use for git bugzilla integration:
> >> http://search.cpan.org/~bmc/WWW-Bugzilla-1.5/
> >>
> >> Regards,
> >> Markus
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/libreoffice/attachments/20120619/765ac4d6/attachment-0001.html>