[Libreoffice-qa] How to change QA processes: was: What should we do with bugs filed against Extensions/Templates?

[Robinson Tryon]

(Petr touched on some similar points; I haven't had a chance to finish
my reply until today :-)

On Sun, Apr 7, 2013 at 4:22 AM, Rainer Bielefeld
<LibreOffice at bielefeldundbuss.de> wrote:
>
> Quality Assurance has to do something with quality, and we have to observe
> and grant the quality of your own work. And here I have serious concerns.
>
> Let's take "Bug 63210 - EDITING: Superscript Defaults incorrect". Joel,
> please excuse me for taking that example, it was the first but I saw this
> morning in my "New bug reports" folder, and it's a typical one for
> observations I see with rising number in the last months. The goal seems to
> be to get the reports out of the QA process as quick as possible (because we
> have so many untouched reports), but I do not like that way, that only
> increases the amount of unfixed confirmed bugs.

I think that QA is best served by striking a balance between accuracy
and speed. We must seek to produce high-quality output, but if we fall
further and further behind each week, then we'll never catch up,
right?

> If we cans save 5 developer
> minutes at each bugs with careful research, that's a potential of 55 hours
> for the 679 bugs what changed to NEW in March 2013. That's 1 work of week
> for a busy developer!

Recruiting more people into the QA process can give us more man-hours
for triage, fdo cleanup, and everything else. Especially with complete
newbies, I agree that a certain amount of preparation and training for
our team members can pay off by reducing the amount of time developers
need to spend getting up to speed or tracking down each bug.

But that isn't to say that we should agonize over each tiny detail of
the bug report. I suggest that we try to cultivate a better
back-and-forth collaboration with developers, so that a dev can either
take a bug report and run with it, or can toss it back to QA and ask
for more info, better repro steps, more test documents, bibisection,
or whatever else would help them to use their time more efficiently.

> It has status NEW, what means "ready for bugfixing",
> <https://wiki.documentfoundation.org/QA/BugTriage#Step_5._Set_Status> item
> 1, but I believe it's not:
>
> a) We have a report without info concerning Version and OS, and a
> confirmation for 1 Linux distribution. So current knowledge (for the moment)
> seems to indicate a LINUX problem? Should the report leave QA process
> ("NEW") without having this corrected or at least mentioned? Well, I confirm
> that for WIN, and so finally the found OS selection is correct, but without
> reasoning that's simply wrong. Reporter should be asked for related info.

Based on my experience with the Ask site, it can sometimes be tough to
ask the Reporter for additional information and get a useful answer
(or any answer at all). Being able to reproduce a bug on more than one
platform is nice, but if we don't have a very large QA team right now,
and if some people don't have multiple OSes on which to test bugs, I
can imagine that narrowing-down the affected Platforms might be
something we consider lower priority than just getting a first-round
confirmation on our outstanding set of UNCONFIRMED bugs.

Proposal: Is it possible for us to do some of the additional triage
work in subsequent "passes" over the bugs? What I mean is, if we work
quickly to triage and repro a bunch of unconfirmed bugs, is there a
good way that we can later query for bugs that, say, need to be tested
on Windows to determine if they're platform-specific or
cross-platform?

Such a system might help us to keep up with the rapid onslaught of new
bugs while giving us the opportunity to go back and improve older bug
reports as time allows...

>
> b) In between we have 3.6.6.2, what might be the next release. Shouldn't be
> checked whether the problem has been solved in between?
>
> c) This problem is text language related (of course), in German text
> language this auto correct does not exist. That might lead to the roots of
> the problem, I would have liked to find out where that problem started.
>
> d) It should be stated that the problem is limited to <Enter> after the
> date, works fine if you simply type a space or even after <Shift+Enter> or
> <Control+Enter>.
>
> e) Not only Writer is affected, same in Calc with <Control+Enter> and Draw
> (and so probably Impress), might also appear in Base.
>
> f) The summary line still has that meaningless "incorrect". After QA it
> should be "AUTOCORRECT: Superscript of English ordinal number suffixes
> continued behind 'Enter' directly after suffix". Or similar!

Maybe this:

AUTOCORRECT [EN]: Superscript started in ordinal number suffixes
continues to next line

But perhaps even that summary is too technical. Consider the fact that
the bugtracker serves many purposes, one of which is making it easy
for users, triagers, and developers to manage a large set of
outstanding bugs. A large proportion of the people working on these
bugs may not know the difference between 'ordinal' and 'cardinal'
numbers.

Which brings up a good question: Do we expect users to de-dupe their
bug reports, or is that something that QA is happy to do for them?

Sure, QA does a lot of de-duplication, but AFAIK we will still suggest
to users that they look to see if a bug report has already been
filed...

- If the goal of a user's bug search is just to filter out bug reports
that we've received a dozen times, then a more technical summary may
help us to elevate our level of discourse and be very precise with our
bugs.

- If, however, we're hoping that users will find and recognize their
problem in a list of bug summaries, then an emphasis on plain, simple
English becomes much more relevant.

> Queries for DUPs really are painful if you have lots of bugs only
> distinguishable by the words "improperly" and "incorrect" in Summary.
>

True.

- What percentage of our bugs are dupes? (we might have to estimate a bit here)

- What methods do we use to track down and de-duplicate FDO?

- What could we do in FDO or in our triage process to speed-up the
de-duplication process?

> g) Back to versions: This worked fine in 3.3.3, so it seems interesting to
> know where this problem appeared? And it's a regression.
>
> h) The example in the report seems a little misleading to me because of the
> parenthesis around the date. The screenshot shows what reporter typed, but I
> would have mentioned what I typed exactly (well, this is splitting hairs)
>
> i) May be I even would have tested whether the bug persists with new User
> Profile? But I doubt that it's related and so I think I would have written
> "I doubt that it's profile related"

I'm still a bit confused as to the nuances of the User Profile[1].
It's great that we know that resetting it sometimes fixes weird bugs,
but our approach to this problem seems kind of like voodoo at times.
If we are concerned about the data in the profile being corrupted into
an unexpected state, why don't we come up with a plan to verify the
integrity of the profile? Given the number of times that I've seen a
suggestion to reset the User Profile on the Ask site, it sounds like
we could avoid a large number of bug reports though such an
enhancement.

> k) And after having added all this information I would think about adding
> András to CC. I agree with "minor" rating and we should not bother
> developers with too many odds and ends, but on the other hand a known bug
> can be integrated into the work flow for a fix some later.

Right. I wouldn't have known to cc: András in this situation... so
that's an important piece of institutional knowledge.

> l) And I would have thought about an enhancement request for localized list
> numbering with correct localized ordinary numbers, but that already is quite
> a different thing.

True, but it's great to have that mindset when triaging bugs!

> And so back to the beginning. I like to co work with "Swiss wrist watch
> makers", these guys who know for every of their actions at work why they do
> it exactly how they do it, and who always check whether there is a
> possibility how they could do better.

Yes:
-- The watch makers know *why* they do something
-- They are constantly checking to see if they can improve

These concepts are critical to the continuing success of the QA Team,
even as we must expand in size.

> The question is whether there is a place for these people in the LibO
> community. From my point of view, at some places I see too much actionism
> instead of careful work and enhancement (beneath a lot of good work, of
> course), and that's a pain for me. We Swiss wristwatch makers need a
> wristwatch maker workshop with wristwatch makers tools, and I doubt that
> this workplace will be kept in the LibO project.

I think there's definitely a place for the watchmakers in the QA and
the LibreOffice project -- and it's actually a very, very important
one. As we expand the size of our teams, or as we recruit people who
may join us for only a week or a month at a time, we need to
continually improve our processes and document our best practices.

Rainer -- just reading through your description above I learned a lot
about not only the bug triage process, but about how *you* think
through that process. I won't necessarily be a carbon copy of you when
I triage bugs, but hopefully I'll take a lot away from your
walkthrough of a typical bug report.

Getting new volunteers up to speed on how to do QA work in our project
is currently a laborious undertaking. I've been reading wiki pages and
fumbling around in FDO for many months, and I still don't know half as
much as Rainer. I hate to speak like some kind of corporate drone, but
what we really need to do to be effective is to leverage our assets so
that we can improve ourselves, our team, and our interaction with
other parts of the LibreOffice community as efficiently and
beautifully as possible.

What I'd love to do is to try to distill what Rainer and others here
know into more tutorials. Those could be text, audio, or even video. A
user can sit down and read the BugTriage page for 15 minutes, or he
could listen to a narrated screencast walking him through the triage
of a bug from start to finish -- heck, maybe he could even watch the
triage of 2 or 3 bugs in that time. I know that *I* would benefit from
a lesson like that, and I figure that newbies could benefit even more.

Here's a good question: Who knew about the whiteboard status 'TopicQA'
before I mentioned it? That's the flag we're putting on all QA-related
bugs, and it's something that at least the core team should probably
just *know*. But in order for us to know about these things, we need
to make it easy to learn about them. That way we're all on the same
page, and can seamlessly hand off work from one team member to the
next.

> Of course I should have mentioned that I like the efforts to interest more
> people for bug wrangling, the cleanup in the QA wiki, the thoughts ofa more
> structured QA work and and and, but my time slot has been expended with
> laments, so there is no more time for this.

Ah, but you mentioned it, if only briefly, and so we do hear the
encouragement! :-)

Cheers,
--R

[1] https://wiki.documentfoundation.org/UserProfile