[Libreoffice-qa] Test Structure in Litmus

[Petr Mladek]

Rimas Kudelis píše v Čt 17. 11. 2011 v 11:06 +0200:
> On 11.11.16 20:12, Petr Mladek rašė:
> > 2. problem with locale:
> > =======================
> >
> > Imagine the following scenario:
> >
> > 	1. one person do test run in "de" locale; the result is:
> > 		 + 25% P1 functional (lang-independent) tests finished
> >                  + 25% P1 l10n (lang-dependent) tests finished
> > 	2. other person start test run in "fr" locale
> >
> > Result:
> >
> > 	+ "fr" person see:
> > 		+ 0% P1 functional tests finsihed
> > 		+ 0% P1 l10n tests finished
> >            => does all tests again
> >
> > Expected Restult:
> >
> > 	+ "fr" person see:
> > 		+ 25% P1 functional tests finished
> >            	+ 0% P1 l10n tests finished
> > 	   => continue with other functional tests and repeat the l10n
> >               tests
> >
> > By other words, the functional tests are duplicated inside one test run
> > for each locale; the l10n tests are currently duplicated even twice
> > (once by the groups, once by the locale in the test run)
> 
> Sorry, I don't quite get this paragraph above. I think I get the problem
> from our chat yesterday however.

I mean that the tests are done when testers see 100% tests finished in
the "run tests" dialog. The percentage currently depends on the selected
locale. So, people need to do all tests in in each locale to see 100% in
each locale => functional tests are done "num-locales times" because
they see 0% in each locale. l10n tests are done "num-locales times
num-locales" because they are duplicated in the subgroups. And testers
see 0% in each subgroup in each locale.

For example, let's have one functional test, one l10n test, and 10
locales. The current scheme and test run approach causes that the
functional test is done 10 times (instead of 1 time), the l10n test is
done 100 times (instead of 10 times).

> 
> > Possible solutions by Rimas a me:
> > ---------------------------------
> >
> > 2. Remove the locale setting in the "run tests" dialog and ignore it as
> >    we suggest to ignore the "build id". Note that the l10n tests are
> >    duplicated in the subgroups:

> This is indeed very close to what we have now and could be considered as
> a possible temporary workaround. It doesn't sound nice in the long-term
> though.

I agree that this is the easiest and probably the beast solution if we
have troubles to modify Litmus for our needs.

> > 3. Do some more changes in Litmus (suggested by Rimas):
> >
> >    a) add extra checkbox into the test case edit dialog (os somewhere);
> >       it will mark the test case as language specific or language
> >       independent
> >    b) count the statistic of finished test cases according to the
> >       check box; "locale" will be ignored for language-independent
> >       tests;
> >    c) allow to transparently localize test cases => you will see
> >       different text in different locales (can be done later)
> >    d) show statistic of finished l10n tests per locale on a single page
> >       (can be done later)

> I of course agree that it's the cleanest solution. But I'm not sure how
> much time and skill I'll have to implement this.

Let's give it a try. We might also ask for help on the developer mailing
list.

>  In any case, I think
> I'll try to at least add that checkbox rather sooner than later, that
> would be a good start  already. :)

yup :-)

> I have one small question lingering on my mind though: is it better to
> add that checkbox to testcases or some higher hierarchical component
> (subgroup, group?) I'm quite sure a testcase makes most sense, but just
> want to check with you guys.

I am going to answer this in the Yi Fan's reply.


Best Regards,
Petr