Cleaning bug list

[Petr Mladek]

Hi bfo,

this are interesting questions. I put back QA mailing list into CC
because there people there are interested.

bfo píše v Út 19. 06. 2012 v 11:24 -0700:
> This is a very nice workflow, but I have some questions:

> - how you define "Bug prevent users from making professional quality work?"

By "professional quality work", I understand that everything works as
expected. It means that all menu entries, buttons, and dialogs does
something reasonable or something that is described in the
documentation ;-)

These bugs are about all functions that are used even by power-users. If
they do not work at all in reasonable scenarios, they should have higher
severity. If they basically works but they do not have ideal results,
they should have lover severity.


> Interoperability issues are a big obstacle. If this package is supposed to
> read/write MSOffice files, then every new bug in this area should have high
> priority by default.

Sure. On the other hand, it would be ugly if the application is able to
perfectly import documents but you could not modify them because of poor
or buggy functionality. We need to ballance it with functional bugs.


>  I have reported two annoying:  
> https://bugs.freedesktop.org/show_bug.cgi?id=47138 47138  and 
> https://bugs.freedesktop.org/show_bug.cgi?id=49102 49102 . Are those
> "prevent users from making professional quality work" enough?

Sure. They are perfect candidates.


> Home users can live with a workaround, but it is impossible to expect
>  that xx(x) users will do it xx times a day. Everyday. That is not
>  "professional" (even worse - it is very "unprofessional" for those
>  users migrated from MSO, where it just worked).

No it is not professional. On the other hand, we need to make difference
between broken and non-optimal functionality and also with the level of
annoyance. If would be ugly if most bugs ends here and the "minor" and
"trivial" categories are empty ;-)


> I'd change the workflow a little bit by putting the obvious things at the
> top:
> - feature requests aka wishlist

I do not have any strong opinion for this. I think that it is is good to
be able to discuss features, so "enhancement" bugs in bugzilla might be
usable.


>  - add target milestone if a feature is planned already

This must be done by developer, anyway :-)


> - bugs reported against not supported branches - exclude those at reporting
> level by disabling old versions in select fields; but what to do when they
> appear anyway -  mark them as INVALID at triaging? ask the reporter to check
> in new version? Any comments?

We need to be more careful here. The current rule is to do NOT modify
the version picker if you reproduce the problem with newer version. It
is because it is very valuable to know how the bug is old. It helps
developers to locate it. So, maybe, we should do it the other way and
set the field to the oldest version where it was found.


> - is it Most Frequently Reported - then just dupe it 
> and now, if a bug is still there, triage it by the proposed workflow.

What do you mean by "Most Frequently Reported"?

Note that Joel's flowchart was about prioritizing. It is only one part
of the bugtriage process. Yes, looking for duplicates should be done
before.


> - how you define most and many - I reported those two bugs on behalf of ~xx
> people - is my bug better than other reports? Should this be controlled by
> number of dupes? At what levels? Maybe enabling voting feature of Bugzilla
> would help here...

Good question and hard to answer. The number of dupes is a good helper.
I would personally enable voting as well but some other people thing
that it is not objective. Otherwise, I use a common sense. I try to
guess how each function is hard to use and how many people would do such
action.

Most people do opening documents, do simple editation (elements from
toolbar), saving, printing.

Many people does more professional things, like using styles, adding
footnotes, inserting table of content, do computation in Calc.

Only experienced users use macros, work with databases, send mails using
addressbook...

Another good symptom is how is the function hidden in menu ;-)



> - regressions - this is a major problem and I think it deserves its own
> place in the workflow. It is not a big problem for home users to switch the
> version back and forth. But when we are talking in terms of bigger
> deployment, when you have xx computers (and growing) it becomes a real
> blocker sometimes. What good are new and fixed versions for, where there is
> a regression which disqualifies the software for daily use? That is a big
> problem.

You are right, regression are bad and we need to fix them with high
priority. On the other hand, you still need to compare them with other
reported bugs and decide what is more annoying for the daily work. We
could not blindly set the highest priority for a tiny functional problem
just because it is a regression. As someone said, you could view almost
any bug as regression, so we need to prioritize them as well.

I would keep the current approach. Add "regression" into Whiteboard and
set one level higher priority that you would set for non-regression.


> - backtraces - not all people can do proper backtrace, especially on
> Windows, so maybe while triaging crash bugs one could ask for one a QA
> member? BTW: Having own dev build of LO 3.5.4.2 I can be QA contact for
> missing bt for all Windows crash bugs (except Base) on that branch. 

If you could provide backtrace for some bugs, it is great. 

In general, we do not have enough QA people who could triage bugs, so we
should leave more work for reporters as well. We just need to make it
easier for them to provide the backtrace.

It actually helps to prioritize bugs as well. If reporter is not willing
to provide more information, the problem is probably not that important
for him.

Also note that some crashers are reproducible only in very strange
circumstances. Also some scenarios are very hard to prepare. We just
need help from reporters in these cases.


> - permissions - seems like every user has canconfirm (Can confirm a bug) and
> editbugs (Can edit all aspects of any bug). Did you think about changing
> that, so one could not interfere with QA actions or vandalize the bug
> reports?

AFIAK, it is better solved in new bugzilla release and we might end
there sooner or later. I wonder if it is a real problem now. There
always will be people that will do problems, for example by adding
annoing comments.

If someone change the priority a wrong way, I would ask them not to do
it, explain that the level is not that important, and change it back. If
they change it once again, I would leave it as is. Developers would
filter it themselves :-)


> Bugs triaged.  But those are only b.f.o. bugs. Checking release info for
> 3.5.5.1 I can see such acronyms like bnc, i, rhbz and there are bugs
> reported on b.f.o. coming from aoo (where they may be fixed already). Not
> mentioning ubuntu, gentoo, opensuse and other bugtrackers. This is a very
> serious problem, because not all bugs are in one place. Any comments about
> that?

Why do you think that it is serious? :-)

The release mentions the list of bugs with links to the different
bugzilla trackers, see
http://wiki.documentfoundation.org/Releases/3.6.0/Beta1 so it is
relatively easy to find them.

I think that it is not worth spending resources on migrating bugs to a
single bugzilla. IMHO, it is better to spend time on fixing other bugs,
improving the functionality, testing, ...


> Does the developers have their workflow in Bugzilla? I compared few commits
> (libreoffice/core libreoffice-3-5-5) with b.f.o reports:
> - 50603 - not assigned, UNCONFIRMED>RESOLVED FIXED
> - 43967 - assigned, UNCONFIRMED>NEW>RESOLVED
> FIXED>REOPENED>ASSIGNED>RESOLVED FIXED
> - 48601 - not assigned, UNCONFIRMED>NEW
> - 50988 - assigned, UNCONFIRMED>ASSIGNED>RESOLVED FIXED
> - 43249 - assigned, UNCONFIRMED>NEW>RESOLVED FIXED 

The ideal process is described at
http://wiki.documentfoundation.org/BugReport_Details#Initial_state
Of course, developers are just humans, sometimes quite overloaded and
they do not follow the process.


> So, not bad but if there is more not assigned RESOLVED FIXED

The assigned filed is currently only useful to avoid conflicts between
developers. There are only volunteers in the project, so it can't be
used to force anyone to solve bugs. The one who fixed the bug is usually
in CC, so it is enough to add comment, you need not add any name or so.


>  or NEW bugs with commits in the repo

Sometimes the fix is only partial and some more work is needed.
Sometimes developer just forget to close the bug. If you are in doubts,
feel free to add a comment to the bug. It will teach the developer to
close the bug himself ;-)


> , then I see a problem here. QA should know what is
> happening with the bugs without the need to follow the comments. How does it
> look for xxxx commits?

I see the pain. Well, we are just humans. As I said, you might teach
developers to close bugs by asking about the state. I wonder, how often
do you see such a bug with commits that is not closed?

BTW: It has also the other side. It is great when bugtriagers provide
correct and clear comments. Developers sometimes have hard times to find
the relevant information as well :-)


>  Also I see a lot of commits without bug report number
> in it. What is fixed then? I see it as another problem.

Well, we do not want to create bug report for each fix. It is too
complex process with poor results. If a developer is talkative, she
provides a good commit message. If she is not talkative, she would
create useless bug report. We could motivate them by asking for details,
saying thanks for info, ...

Anyway, I think that it is not important now. We do not have enough
people who could test all commits, so we do not need perfect commit
messages.


> I see you do not use QA field in Bugzilla. Why is that? Every component of a
> product can have its default QA specialist. Then developers responsible for
> a component quality can get a notice of new bugs instead of news about all
> of them. Also QA specialist assigned here could be responsible for
> bibisecting, bt delivery or verifying the bug.

We currently do not have enough people doing bug triage. If we have, we
could start thinking about specializing. 


> By the way - Browsing through dev/qa mailinglist I can see that you prefer
> those lists than bugs in Bugzilla. For instance - are those action items
> from ESC or QA meetings reported as bugs in Bugzilla? b.f.o is not used
> project wide or am I missing something?

bugzilla is good for tracking bugs because you could add important
people into CC, could add attachments, ask others, have all comments in
one place, could assign it, close it, ...

Many tasks on the team meetings are not related to the code, for
example:

        + Late feature reviews (Kendy, Astron, Eike, Markus)

		these people should just communicate with each other
                because of something

        + icons: send out some mail addresses to poke (Astron)

		it is about providing list of E-mails to another person
                in the ESC-team

        + post on-line update numbers for 3.6 testers (Kendy)
                + 600 on-line B1 / updaters registered
                + promote B2 a bit more ...

		it is about providing some numbers to the team
		also it is about creating a plan

I do not think that bugzilla is a good tool for handling these type of
tasks ;-)


> Do you VERIFY bugs? I see just 62 bugs as VERIFIED FIXED. I do not see such
> section in Getting Involved. Surely there are users willing to participate
> this way.

We do not have enough people to sort new bugs, so nobody took care of
this :-)

Well, it might be good idea to add this into the "Getting involved"
section. It might bring some new people that might be later converted
into triaging bugs or doing other QA tasks. Would you like to create a
wiki page describing the process? IMHO, it is should explain:

	+ how to find bugs to verify (priority)
	+ how to find builds (releases, daily bugs)
	+ how to mark the bug (add comment?)


> I see you do not use flags and prefer Whiteboard. Why is that? Maybe a
> better idea than a MAB nightmare (IMHO meta bugs with >300 comments are
> useless) would be a release tracking flag, where QA would like to see a fix.
> Mozilla projects use them a lot for instance...

What do you mean by the flag?

MAB just works. Nobody reads the whole history. The point it is that the
developers mailing list is in CC. Hence any new comment is sent there
and developers are aware of new bugs. This is why it is important to add
useful comment.

Alternatively, you just look at the query
https://bugs.freedesktop.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=NEEDINFO&field0-0-0=blocked&product=LibreOffice&query_format=advanced&type0-0-0=anywordssubstr&value0-0-0=37361&order=bug_severity%2Cpriority%2Cbug_id&list_id=86968
and you see list of still opened bugs.

Note that the queries are available at
http://wiki.documentfoundation.org/Release_Criteria#Blocker_Bug_Nomination


> Well, sorry for such a pack of off-topic questions, but I'd like to
> understand QA in this project better.

It is fine. I hope that I did not discourage you. I feel that you want
to have everything perfect which is great. On the other hand, we have
only limited resources, we are just humans, and we need to optimize the
processes for this. The target is clear. We all want to improve LO with
each release and have the best office suite ever :-)

Thanks for all your contributions.


Best Regards,
Petr


PS: If you reply, please try to configure your mail client, so it puts
some prefix "> " before the old text and put your answer inline. Please,
do not use top-posting! It helps to keep kontext in such long mails with
many questions :-)