[Libreoffice-qa] Test case naming

[Petr Mladek]

Rimas Kudelis píše v Út 15. 11. 2011 v 13:43 +0200:
> Hi all,
> 
> I took a quick look at how testruns are created and what we can and
> cannot choose when creating a testrun. My suggestion is based on the
> fact that each testrun comprises of
> 
> * all subgroups in
> * one or more testgroups in
> * exactly one branch in
> * exactly one product.
> 
> That is, a testrun cannot span through multiple branches or products,
> nor can you specify which subgroups to add to it. You can only choose
> groups.

This is important information. Thanks for looking at it.

> > Overall satistics:
> >
> >     We now have 3 branches:
> >
> >         master regression branch
> >             - 1 group
> >                 - 7 subgroups in each group
> >         master l10n branch
> >             - 4 groups
> >                 - 7 subgroups in each group
> >         master feature branch
> >             - we may not consider case priority here?
> 
> Current scheme does not allow creating a catch-all testrun comprising
> all tests for a single product/branch. I don't know whether or not such
> ability is desired, but it looks logical to me.

You are right. 1 run is the best solution. Well, 2 or three test runs
for a release still might be acceptable.

>  Also, maybe I shouldn't look that far into future, but I hope that
>  there will come a day when proper localization of testcases will be
>  possible (that is, instead of creating a clone of testcase X in
>  another language, we would actually be able to translate testcase X
>  into that language). With that in mind, current testgroups (which
>  represent different locales) would become unneeded.

I am not sure if the l10n tests will be pure word-to-word translation.
Some things are done completely different in different languages, for
example the layout of the letter template, right-to-left language
features, decimal number delimiter, dates. I am sure that some
languages would need special tests.


BTW: How is the localization used during test run?

I know that I select locale in the "Run Tests - Your Chosen Test Run"
dialog, see
https://wiki.documentfoundation.org/Litmus/Litmus_User_Guide#Enter_Test_Configuration

If I select "de", will I see only the "de" test cases from the l10n
branch? I am not sure if it is important but it would be nice.


> > Case 1. Carry priority information with subgroups name

> This scheme would make it impossible to create a testrun for e.g. just
> P1 tests. Or just writer tests. I guess this is pretty much the same as
> what we have now, so all criticism from above paragraph applies here too.

Good point. We can't use this.

> > Case 2. Carry priority information with groups name

> This is probably OK if you don't mind having to create three testruns
> for each version

It is not ideal but acceptable.

>  and be unable to create testruns for separate components

IMHO, this is fine.

> . I think you forgot locales here though. I assume that for
> each locale/priority there would be a separate testgroup?

I guess so. Hmm, that would create too many test groups. See my other
mail.


> > Case 3. Carry priority information with branches

> IMO, Case 3 is the worst case, because for each version of LibreOffice,
> you would have to create 4*3=12 testruns – one for each priority/test
> type combination.

I am not sure if it is the worst case because the solution 1 was bad as
well :-) Anyway, I think that 3 is not a way to go.

> My suggestion is to have a single branch, but carry Priority, Locale and
> Component information in the Testgroup, and to represent test type by a
> subgroup. That is, what is now a branch, would become a subgroup, and
> everything else would become a testgroup (see the attached PDF file).

Hmm, I am afraid that we would get too many testgroups. It produces 7
(General, Base, Calc, Draw, Impress, Math, Writer) testgroups for one
language. We have 109 localizations in sources => we could end up with
more than 700 test groups.

> This allows to:
> * create testruns based on priority, locale, component, or any
> combination of these

The question is what test runs we will want to create.

> * create a single catch-all testrun for a single version of LibO

would be nice

> * share the same General Functional Tests subgroup between testgroups
> designated for different locales (that is, you would create this
> subgroup once, and add it to all locale groups)

I am not sure what you mean with this. Could we share subgroups between
test groups in Litmus? Is it clearly visible or is hard to maintain and
see?

BTW: What do you mean with the "Basis Functional Test"? We do not have
this subgroup in the current structure.


> * drop 75% of the testgroups (there would be about 28 of them initially)
> if/when proper testcase localization is implemented, still not rendering
> the remaining testgroups useless.

I am not sure if we really could drop them. Well, the question is if we
need to split between lang-dependent and lang-independent on the top
level.

Heh, it seems that it is quite complex problem. I am going to write
another mail where I will try to summarize some ideas :-)

Thanks a lot for the valuable input.


Best Regards,
Petr