[Libreoffice] Libreoffice QA process outline - a coarse draft.

[Petr Mladek]

Sophie Gautier píše v Po 11. 04. 2011 v 14:59 +0300:
> Hi Yifan,
> On 11/04/2011 13:59, Yifan Jiang wrote:
> >     >  13:25<  sophi>  yifan: we distinguish between l10n and native language teams,
> >     >  it's not the same members (even if it is the same language)
> >
> >     Would you help distinguish native language teams and l10n team? Do they
> >     have overlapped work between each other? Do they have overlapped work
> >     with pure function test? Thanks!
> 
> It depends of the size of the team in fact. For large teams like French 
> or German, localizer and QA members are not the same persons. I'm doing 
> the localization, where a team of about 10/15 members is doing QA and 
> doesn't care about the localization process. If the language team is 
> small, the members doing localization, QA, even documentation will be 
> the same persons.

> > [Release QA Process - Localization]
> >
> >     - I am not sure about the l10n process, but as a tip, information from
> >       Petr shows that current l10n process might be optimized by the fact all
> >       LO localizations are built at the same time, so most functionality just
> >       work the same.

If I understand it correctly, there is not that big overlap in the
testing done by l10n teams. It seems that it is oriented on validating
strings, so it needs to be done for each localization by native
speakers.

The test duplication might be in the testing done by native language
teams. Of course, there are lots functions that depends on the
localization, e.g. spellchecking, autocorrection, default units, date
formats, decimal separator, Asian, CTL support. Though, most of the
functionality should be language independent.

Or are the native lang teams concetrated only on this lang-dependant
functionality?

> >     - Moreover if there is some overlapping between native language testing
> >       and function testing. We may want to optimize them as well.
> >
> >     - TBD...
> 
> We need a special build called KeyID build for l10n QA. This build 
> contains a unique ref for each string in the UI, see
> http://wiki.services.openoffice.org/wiki/KeyID_Build

Do you need the KeeID build for every localization?
Do you need it for every beta and rc?

> > [Release QA Process - Function test]
> >
> >     - The latest Release build (RC) for testing should be available at:
> >
> >         http://www.libreoffice.org/download/pre-releases/
> >
> >     - For each release, a test run will be created in Litmus TCM
> >       tool(http://tcm.documentfoundation.com). QA would need to run them in at
> >       most 10 days after the build ready (based on Release Criteria)
> >
> >       (howto litmus http://wiki.documentfoundation.org/Litmus)
> >
> >         + Who're going to create test run?
> >
> >         + When should the test run ready?
> >
> >         + Who're going to define test cases?
> Hum, we should try to set a team dedicated to the test maintenance, 
> enhancement and organizing the localization.

+1

> >     - QA announces RC build pass/fail based on Release Criteria
> >
> >       http://wiki.documentfoundation.org/Release_Criteria
> >
> >         + Where do we announce the results (mailing list, wiki)?
> Collect the results on the wiki may be more easier for every body to 
> participate.

It might be enough to show them in Litmus if they are well visible
there.

> > [Generic Libreoffice QA Process]
> >
> >     QA is being done at any time from develpment phase to final release. This
> >     section aims to provide a guideline to unify those testing related actions
> >     to be followed.
> >
> >     - Blockers Nomination process
> >
> >         http://wiki.documentfoundation.org/Release_Criteria
> >
> >     - Bug report/triage process
> >
> >         http://wiki.documentfoundation.org/BugReport
> >         http://wiki.documentfoundation.org/BugTriage
> >
> >         We may want to add sections in Bug report/triage pages:
> >
> >         Regressions
> >
> >             The bugzilla has a 'regression' keyword in the 'Keywords' field.
> >
> >             Our attitude for a regression problem? I think for the same problem,
> >             the regression problem should have a higher priority than a normal
> >             one.
> +1

I agree. Of course, it depends on the type of regression. If it is in a
feature that has been implemented many years ago, author is not longer
around, it had only few users, ...

> >
> >         Bug Severity
> >
> >            Bug Reporter's responsibility to mark it.
>
> >         Bug Priority
> >
> >            Bug Triager's job - We may need a priority setting guide similar to release
> >            criteria. Do we have it already somewhere?
> 
> I don't think that we have such a guide yet.

Here is the guide for Novell bugzilla. Of course, we would need to
update it for Free Desktop bugzilla and LO but...

=== Bug Severities ===
====Blocker====
* Prevents developers or testers from performing their jobs. Impacts the
development process.
* (Documentation) Key documentation is missing for critical testing and
review.
* (Security) An issue that blocks the completion of an SRB architecture
and/or export review.

Examples:
* Unable to login
* Unable to performance certification tests
* Unable to update system

==== Critical ====
* Crash, loss of data, corruption of data, severe memory leak.
* (Documentation) prescribes or doesn't warn against actions that cause
data loss or corruption.
* (Security) A CVSS base score of 5.0 - 10.0 is a critical defect.

Examples:
* Crash that is repeatable and evident to multiple users
* Memory leaks that lead to OOM errors during average use in one week or
less

====Major====
* Major loss of function, as specified in the product requirements for
this release, or existing in the current product.
* (Documentation) missing, misleading, inaccurate, or contradictory
information to the degree that by following the documentation successful
completion of fundamental tasks is unlikely.
* (Security) A CVSS base score of 2.5 - 4.99 is a major defect..

Examples:
* Prevents mandatory feature from working properly
* Feature regression from previous release

====Normal====
* Non-major loss of function.
* (Documentation) missing, misleading, inaccurate, or contradictory
information in the documentation, but successful task completion is
probable.
* (Security) A CVSS base score of 1.0 – 2.49 is a normal defect.

Examples:
* Prevents important or desirable feature from working properly

====Minor====
* Issue that can be viewed as trivial (e.g. cosmetic, UI, easily
documented).
* (Documentation) contains stylistic or formatting issues, but
functionality is not hindered.
* (Security) A CVSS base score of 0 – 0.99 is a minor defect.

Examples
* String typo

=== Bug Priorities ===

====P0 - CritSit====
This priority is reserved for NTS.  It is not used for defects
associated with products in development.

====P1 - Urgent====
Use this priority for urgent issues

Examples:
* Blocker: Generally is a P1
* Critical: Nautilus crashing while opening a file for all x86_64
installations
* Major: Fingerprint support authenticates regardless of the fingerprint
swipes
* Normal: Package management log does not get rotated (will get large
fast)
* Minor: SLED is misspelled in bootsplash

====P2 - High====
Use this priority for mandatory defects, enhancements, and work items.
That is, for items that  must be resolved in this release.

Examples:
* Critical: Nautilus crashing while opening a file for all x86_64
installations over ssh
* Major: Fingerprint support (mandatory feature) does not work with
gnome-screensaver
* Normal:  Package management system is not able to lock packages with
regular expressions (but rug parity is needed)
* Minor: Notification about potential security issue is obscured on
screen

====P3 - Medium====
Use this priority for desirable defects, enhancements, and work items.
That is, for items we would like to fix, but we won't hold shipment for
them.

Examples:
* Critical: Nautilus crashing while opening a file ssh for certain
non-default configurations
* Major: Fingerprint support (mandatory feature) does not work with sudo
* Normal: Package management system is does not display correct progress
* Minor: Notifications do not wrap text properly and can be cut off
sometimes

====P4 - Low====
Use this priority for optional defects, enhancements, and work items.
This priority is not as strong as desirable.

Examples:
* Critical: Nautilus crashing while opening a file ssh for particular
user with a provided backtrace
* Major: Fingerprint support (mandatory feature) does not work with sudo
for users with complex configurations
* Normal: Package management system does not show correct icon for
enhancement updates
* Minor: Notifications do not have the correct icon sometimes

====P5 - None====
Indicates that a priority has not been assigned.


Best Regards,
Petr