[Libreoffice-qa] Test Structure in Litmus

[Petr Mladek]

Yifan Jiang píše v St 16. 11. 2011 v 16:38 +0800:
> Hi Petr,
> 
> Thanks for the detailed summary, I hope I can review this before sending
> mine:) If I understand you correctly, scheme D ignores a requirement, doesn't
> it?

What requirement is ignored by the scheme D?

> > 7. Test run creation:
> > 
> > 	+ requirements:
> 
> > 		+ create test run for a given component (only Writer
> >                   modified)
> 
> In the latest mail I just replied to "Test case naming" I shared a similar
> idea. I can see the main differences of ours focused on 2 points:
> 
> 1. Do we need components split in L10n tests?
> 
>     I have ever considered the leraning curve when l10n and function test
>     doesn't have a consistent scheme. But from reconsidering, it is probably
>     natural for a user to see such language split when clicking a
>     Localizatoin test group. Plus it reduces the test group size
>     significantly, which is my favorite as well. So I give up this concern :)

Good. I hope that we will have only few language dependent test cases. I
think that it is not work splitting them by component (general,
writer, ...). We do not want hunderts of subgroups with a single test
case.

>     Another question is still about how many cases will we have. One thing I
>     didn't mention is with this scheme, when we create a test case we need
>     to review all the cases to make sure they were not created
>     before. Currently we only have 23 cases, it is okay if we review all of
>     the cases summaries when creating a new one. But if we get even more cases (> 50
>     for example), it becomes painful and hard to manage.

Good question.

So, the testtool had more than 2000 test cases. I think that most of
them made sense. Of course, it is not realistic to duplicate testtool
effort manualy. On the other hand, I am afraid that it would be hard to
get support for the testtool because developers do not like it, so what
we could do.

My idea is to create as many reasonable manual test cases as possible.
In the long term, we should try to get support from development and
automatize them step by step.

How to deal with some many test cases:

	+ the structure in Litmus is important:
		+ must be clear and relatively easy
		+ the number of test groups, subgroups,
                  and test cases in subgoups must be well balanced
		+ test case must have a good title, so the content is
                  clear just by reading the title 

BTW: I do not like the current test cases. They are too trivial. For
example, we do not need an extra test cases for creating new document
in Writer:

	+ it is too much paper work for trivial operation
	+ this operation is done for all components (Writer,
          Calc, ...) by smoketest withing few seconds
	+ this operation is an atomic part of many other test cases

I suggest to take inspiration in the testtool and create more reasonable
test cases.


> 2. Do we need an extra feature branch?
> 
>     It is pretty fine for me to have another branch, I did actually
>     prefer to leverage branches in a reasonable scale :)

good.

I still thing that the variant D best fits our needs :-)

> > D. 2 branches, 8 testgroups, 7/100 subgroups:
> > 
> > + LibreOffice product (two branches per release):
> > 
> > 	+ master branch (8 test groups):
> > 		+ P1 - function testgroup (7 subgroups)
> > 			+ General
> > 			+ Base
> > 			+ Calc
> > 			+ Draw
> > 			+ Impress
> > 			+ Math
> > 			+ Writer
> > 		+ P1 - l10n testgroup (100 subgroups)
> > 			+ en
> > 			+ de
> > 			+ fr
> > 			+ ...
> > 		+ P2 - function tesgroup(7 subgroups)
> > 			+ General
> > 			+ Base
> > 			+ ...
> > 		+ P2 - l10n testgroup (100 subgroups)
> > 			+ en
> > 			+ de
> > 		+ P3 - function tesgroup(7 subgroups)
> > 		+ P3 - l10n testgroup (100 subgroups)
> > 		+ P4 - function tesgroup(7 subgroups)
> > 		+ P4 - l10n testgroup (100 subgroups)
> > 	
> > 	+ 3.5 regression branch (9-12 test groups)
> > 		+ P1 - function
> > 		+ P1 - l10n
> > 		+ ...
> > 	+ 3.5 feature test branch
> > 		+ ???
> > 	+ 3.6 regression branch (8 testgroups)
> > 	+ 3.6 feature test branch
> > 	+ ...

Best Regards,
Petr