[Libreoffice-qa] Updating Litmus testing organization.

[Yifan Jiang]

Dear Libreoffice QA fellows,

Looking through the past year's knowledge of our Libreoffice Release QA and
people I talked in Libreoffice Conference, I was working on some change of
Litmus test cases organization these days.

The target of these works is mainly to make the test case categorized in a way
of more clear and efficient to run. The change is not significant to break our
previous experience but make them more clean.

** Concretely, in short, we are going to divide the single Regression test
branch into Function Regression Test branch and L10N Regression Test branch.

The example of test runs I created from this updated method can be seen:

    https://tcm.documentfoundation.org/run_tests.cgi

Please don't be scared about the following long description, you will only
need to review them if you want to know more details about contributing
writing test cases. On the other hand, you don't have to read them if you are
only interested to run test cases as you did before.

Old test case organization
==============================

The old strategy of orgnizing cases in Litmus can be described as easy as:

+ Branch Master Regression Test
    - Group: en/de/fr/pt-BR
        - Subgroup: base/calc/writer/impress/draw
            - Test Cases (described in the corresponding native language of Group)

So in the master branch of test cases repository, we have 4 Groups, in each of
the group we have 5 subgroups, which contains the substantial test cases. It
is easy to understand why the subgroups are divided by Libreoffice function
components. On the second level of the hierachy, we also have created the
lanugage Groups for the purpose of describing test cases in different
languages so that:

    1. QA from non-english speaking environment can understand the tests in
    his/her native language

    2. Test cases can be modified for a particular language l10n test. For
    example, considering the same test coverage of translation, the wording
    and expecting results canbe slightly different between fr and pt-BR.

Why Divide
==========

Typically, the old strategy brings us a "duplicate effort of execution
problem" when function tests are populated into Litmus. Let me show a good
example of a Writer test cases (w001):

The case in en Group:
#EN-w001 - Creating a new text document
    https://tcm.documentfoundation.org/show_test.cgi?id=7

The case in fr Group:
#FR-w001 - Création d'un nouveau document texte
    https://tcm.documentfoundation.org/show_test.cgi?id=46

The case in de Group:
#DE-w001 - Ein neues Textdokument erstellen
    https://tcm.documentfoundation.org/show_test.cgi?id=87

It can be easily seen they are all describing exaclty same tests of "Creating
a new text document", and they were tested for several times in the different
language QA people (thanks for the effort to test!). However the test itself
is irrelevant with languages. That is to say, for this kind of tests, no
matter in which language, we only need to spend one time (not three or more
times) to execute for each release candidate build.

So it would be nice to treat those cases in another way, which brings our new
branch:

Branch Master Function Regression Test
======================================

Function Regression Test Branch stores language neutral function cases
regardless in which locale the tests will be executed.

For the sake of language diversity of QA people, it would be best to have them
described in different languages as well. Ideally the same case can be shown
in different language based on QA machine's locale or QA's flavor, but it
needs some deep hackers to expand Litmus capacity (see: point 4 -
http://wiki.documentfoundation.org/Litmus_TODO). So as a rude workaround,
currently we just merged all of the language of test description into a single
test case. Therefore no mattter who executes the case, all other people can
see its latest status now.

This is how #w001 looks like now:

    https://tcm.documentfoundation.org/show_test.cgi?id=1054

Specifically the organization of the branch canbe described as:

+ Branch Master Function Regression Test
    - Group: Function Test
        - Subgroup: base/calc/writer/impress/draw
            - Test Cases (each testcase is described in multiple languages)

As you can see we have only one Group called function test here, we do not
have functional overlapped test cases describing in different language anymore
in this branches.

Meanwhile, we can not deny the considerable needs of testing Language specific
features and translations. Actually the old way of organization fits these
requirement very well, so we can just keep what we have in a separate branch:

Branch Master L10N Regression Test
==================================

L10N Regression Test Branch stores language specific test cases which should
all be language sensitive. That is to say, we only put those cases really need
test in multiple languages into this branch. For example: dictionary, spell
check, hyphenazing, menu/dialog translation, asian language layout etc.

Specifically the organization of the branch can be described as same as the
old way:

+ Branch Master Regression Test
    - Group: en/de/fr/pt-BR
        - Subgroup: base/calc/writer/impress/draw
            - Test Cases (described in the corresponding native language of Group)

The English version is still there because we also need to provide a
"synchronizing center" to share information between different
languages. Ideally it is preferrable to have an English version of cases for
each language specific test cases, so that QA who knows English can be
enlighten when he/she creates test cases in his/her own language. Naturally,
it is easy to figure out the test cases in English group will not be added
into any test runs since they are supposed to be only usable in L10N
environment.

Current Progress of Updating
=============================

So the current status of my work is finished to merge all of the test cases
into "Branch Master Function Regression Test". So it is preferred any of the
new Function test cases will go to this branch.

While I didn't remove any old cases at the moment, they are all currently in
"Branch Master L10N Regression Test", since some of the test cases are
re-usable and just there waiting for updating. I just do not have enough
knowledge to update all of them :) What I will do is to update English version
of test cases to make sure it contains language specific test cases only. It
would be nice they will be translated to corresponding language cases, and we
need more L10N specific test cases!

Moreover, I'll also send a mail to admins about how to do all of this, as well
as updating wiki pages.

Please let me know if you have questions or would like to contribute efforts
to any of these! Discussion of the test cases organization is also open :)
Thanks for your attention and never hesitate to contact with me if you have
any questions!

Best wishes,
Yifan