[Libreoffice-qa] Test Structure in Litmus

[Petr Mladek]

Yifan Jiang píše v Pá 18. 11. 2011 v 13:17 +0800:
> Thu, Nov 17, 2011 at 11:06:43AM +0200, Rimas Kudelis wrote:
> > > 2. Remove the locale setting in the "run tests" dialog and ignore it as
> > >    we suggest to ignore the "build id". Note that the l10n tests are
> > >    duplicated in the subgroups:

> > > 	+ it will be hard to see how many l10n test cases were finished
> > >           in the various localizations; you would need to enter the
> > >           "run tests" dialog with different setting
> 
> Just found an alternative way is "Reporting -> test runs" can show it more
> easily. Instead of percentage of finishing, detailed execution record
> of each test case in group/subgroups are summarized there.

Yup, this might be usable.

> > This is indeed very close to what we have now and could be considered as
> > a possible temporary workaround. It doesn't sound nice in the long-term
> > though.
> > 
> > > 3. Do some more changes in Litmus (suggested by Rimas):
> > >
> > >    a) add extra checkbox into the test case edit dialog (os somewhere);
> > >       it will mark the test case as language specific or language
> > >       independent
> > >    b) count the statistic of finished test cases according to the
> > >       check box; "locale" will be ignored for language-independent
> > >       tests;
> > >    c) allow to transparently localize test cases => you will see
> > >       different text in different locales (can be done later)
> > >    d) show statistic of finished l10n tests per locale on a single page
> > >       (can be done later)

> > I have one small question lingering on my mind though: is it better to
> > add that checkbox to testcases or some higher hierarchical component
> > (subgroup, group?) I'm quite sure a testcase makes most sense, but just
> > want to check with you guys.
> 
> IMHO, attach it to test case is the most safe way from the design's view,
> which gives us the most flexibility of expanding the system.

I slightly prefer this because of the flexibility. We could switch the
meaning of any test case at any time without moving it in the structure.
It would help us to create easier structure.


> But in UI level, subgroup maybe a better place because the test cases will be
> created by different people, some of whom may not notice the usage of the
> checkbox for each test case.

It might be solved by reasonable default. I would set it as functional
test by default because most tests should be language independent. It
should be possible to fix the status at any time, see below.


>  Once the checkbox is checked in an inappropriate
> manner, it will be hard to manage and the test statistics is potentially
> calculated in a wrong(or tricky) way.

I hope that the test results are written into a database and the
statistics are calculated on demand. If this is true, the check box
would modify how the test statistic are be calculated but it would not
modify the data in the database. So, the change in the checkbox should
not cause big troubles.


>  Keeping the checkbox in a subgroup could
> be more controlable because of the stability of subgroups contents and
> structure. Test case authors would only need to put their new test cases in
> the correct subgroup like we are doing now and nothing else.

The question is if we really want to split the language-dependent and
language-independent tests by test groups or subgroups. It creates more
complicated structure. For some people, it might be hard to decide the
right group.


Best Regards,
Petr