[Libreoffice-qa] Test Structure in Litmus

[Petr Mladek]

Petr Mladek píše v Út 15. 11. 2011 v 19:44 +0100:
> 4. platform:
> 
> 	+ why:
> 
> 		+ some tests might depend on platform; for example the 
>                   open/save dialogs look different way on Linux,Windows,
>                   and MAX; also KDE vs. GNOME
> 
> 	+ main requirements:
> 
> 		+ these tests must be done on all affected platforms
> 		+ test results must clearly show where it was tested

[...]

> b) platform is not much important
> 
> 	+ there will be only few platform dependent test cases
> 	+ I am not aware about any test that would be platform 
>           and language dependent
> 	  => there will only platform dependent functional tests
> 	
> 	  We could mention the requirement in the summary or test
>           description.

[...]
 
> v) "Platform" affects only few tescases
> 
>    => mention platform only in test case name

Sigh, my understanding of platforms was not clear. We need to differ two
things:

	+ some tests make sense only on some platforms, e.g. KDE, GNOME
          integration is done only on Linux

          we might solve this be a note in the test title or test
          description; we might quickly skip this test on all other
          platforms


	+ we should do all tests on all platforms because the things are
          packaged different way, there are used different compilers,
          there are lots platform specific hacks in the code; we quite
          often see bugs that happen only on one platform

	  We might solve this by creating separate test runs for
          Linux, Windows, and MAC

          => it gives even bigger pressure to keep the number of
          branches on minimum


I still think that the scheme D could work.

IMHO, it does not make sense to add platform to the structure because
most of the tests should be done on all platforms and we do not want to
duplicate them in Litmus. There will be only few platform-specific
tests. 
=> it is not worth to multiply the number of groups by platform

Note that we probably do not need to create test runs for the feature
branch => there will be only 5 runs (Windows,Linux-i586,Linux-x86_64,
MAC, MAC intel) for the release.


> D. 2 branches, 8 testgroups, 7/100 subgroups:
> 
> + LibreOffice product (two branches per release):
> 
> 	+ master branch (8 test groups):
> 		+ P1 - function testgroup (7 subgroups)
> 			+ General
> 			+ Base
> 			+ Calc
> 			+ Draw
> 			+ Impress
> 			+ Math
> 			+ Writer
> 		+ P1 - l10n testgroup (100 subgroups)
> 			+ en
> 			+ de
> 			+ fr
> 			+ ...
> 		+ P2 - function tesgroup(7 subgroups)
> 			+ General
> 			+ Base
> 			+ ...
> 		+ P2 - l10n testgroup (100 subgroups)
> 			+ en
> 			+ de
> 		+ P3 - function tesgroup(7 subgroups)
> 		+ P3 - l10n testgroup (100 subgroups)
> 		+ P4 - function tesgroup(7 subgroups)
> 		+ P4 - l10n testgroup (100 subgroups)
> 	
> 	+ 3.5 regression branch (9-12 test groups)
> 		+ P1 - function
> 		+ P1 - l10n
> 		+ ...
> 	+ 3.5 feature test branch
> 		+ ???
> 	+ 3.6 regression branch (8 testgroups)
> 	+ 3.6 feature test branch
> 	+ ...

Best Regards,
Petr