[Libreoffice-qa] Test case naming

[Yifan Jiang]

Hi Rimas and Petr,

Thanks for the great discussion :) I appreciate Rimas' new scheme with some
questions remaining. Let me take the opportunity to give a mid term summary
and share my thoughts based on the new scheme.

1. Firstly through the complicated mail thread, I feel it is necessary to
clarify the definition of test types helping us to have a common sense:

    - Function Regression test (Language Neutral Tests)

        It is language neutral and we only need to run it once in any of the
        languages. The localization for this kind of tests could be
        word-by-word.

    - L10N Regression test (Language Dependent Tests)

        It is language dependent and we need to run them in all
        language. The localization for this kind of tests could be
        slightly different language versions based on the libreoffice
        features. 

    - Feature test

        It contains new features testing for each release of
        libreoffice. Test cases are added on demand in each new
        release, and the test cases are not reusable in the next release.

2. In the new scheme pdf, it looks feature tests are not included while a
legacy Basic Function Test is there. Just to make things more clear, I am
udpating Rimas' chart a bit:

     Basic Function Test -> Feature Tests
     General Function Test -> Function Regression Tests

I believe we didn't really have misunderstanding on this but the legacy
problem, right?

3. Do we want components split in L10n test? I don't have a definite answer
for this yet but prone to keep the components. 

On one hand, at least for impress and calc, we have some specific areas to
cover as tempalte, slide show in multiple language, calc cell format etc. On
the other hand, the consistent structure may make people's learning curve more
mild.

4. The amount of how many localization is another concern. In the new scheme,
for each priority we will have "7 components * 4 locale * 4 priority" (112)
groups. If I understand it right, when there are 10 locales, the number will
be 2800 ?!

Image we have 112 test groups in the first section when running test:

http://wiki.documentfoundation.org/images/7/77/Litmus_submit-de.png

While another choice to be considered could be switching the position
of test types and components, something like:

       / p1-en-Feature Tests              - 7 components (en)
       / p1-xx-Feature Tests              - 7 components (xx)
       / p1-en-L10n Regresstion Tests     - 7 components (en)
       / p1-xx-L10n Regresstion Tests     - 7 components (xx)
master - p1-en-Function Regresstion Tests - 7 components (ATM: multi-language case)
       \ p1-xx-Function Regresstion Tests /

Besides it will reduce the groups size to "3*locale*priority", by the fact we
can't select subgroups when creating test run (I didn't notice it before,
thanks Rimas), we can create a test run for particular test types, which might
be more likely to happen than creating test runs for particular components.

It would be more clear of the strategy by comments Rimas' origin plan:

>* create testruns based on priority, locale, component, or any
>combination of these

-> create testruns based on priority, locale, test type, or any
combination of these

>* create a single catch-all testrun for a single version of LibO
>* share the same General Functional Tests subgroup between testgroups
>designated for different locales (that is, you would create this
>subgroup once, and add it to all locale groups)

-> share the same components subgroup in Function Regression Tests testgroups.

>* drop 75% of the testgroups (there would be about 28 of them initially)
>if/when proper testcase localization is implemented, still not rendering
>the remaining testgroups useless.

-> This is the thing I am not pretty sure. Will this switch cause
unhandlable problems for our potential translation system ?

5. Another critical question raised about different testing execution times of
Function and L10N regression.

>> 2011.11.15 17:30, Petr Mladek rašė:
>> I didn't say they have to be translated word-to-word. They should be
>> *localized*, and I would expect a localized testcase to suggest
>> localized number and date formats and other stuff.
>
> I see.
>
> Well another question is how it appears in the test run. The language
> independent tests are proceed once. The language dependent tests need to
> be proceed for every localization.

The test case transaltion can be considered into 2 layers in current test
strategy:

- For Function Regression Tests (Language Neutral Tests)

    Review the definition on the top of the mail, the translation version case
    id is supposed to be the same with its English version. So all of the
    people run the same test cases but read them in different language.

- For L10n Regression Tests (Language Dependent Tests)

    Review the definition on the top of the mail, the translation version case
    id is not the same one as its English version. So all of the test cases
    will be required to run at least once in its own locale.

When localization system is designed, we may need more investigation of them?
Or do I miss anything have been designed?

Those are about all I am thinking, hope they didn't bring more confusion :)
Thanks again for the hot discussion!

Best wishes,
Yifan