[Libreoffice-qa] Test Structure in Litmus
Petr Mladek
pmladek at suse.cz
Tue Nov 15 10:44:59 PST 2011
Hi,
I am going to summarize the ideas from the mail "Test case naming".
I am sorry for the long mail. The problem is not easy. There are many
ideas. I tried to sort them from different point of view. I hope that I
did not miss anything important and it will help to find the final
conclusion.
----------------------------------------------------------
Let's start by a short summary.
Litmus allows to split test cases by:
+ product
+ branch
+ testgroup
+ subgroup
We want to split test cases by:
+ priority (P1,P2,P3,P4)
+ component (General,Base,Calc,Draw,Impress,Math,Writer)
+ meaning (functional, l10n, feature)
+ l10n tests in languages (en,de,fr,it,ja,.............)
+ platform (Linux,Windows,MAC)
We need to take care of the following scenarios:
+ maintenance:
+ creating test cases
+ updating test cases
+ localization
+ branch for new release
+ test run for new release/build
+ doing tests
+ analyzing results
-----------------------------------
Let's look at some conditions:
1. priority:
+ why
+ reduced test run for bugfix releases
+ overal prioritization of the work
+ main requirements:
+ allow to create test run for given priority
+ easily run tests according to priority
+ easily see test results by priority
2. component:
+ why
+ reduced test run if only writer is modified
+ helps to split work by expertize
+ main requirements:
+ allow to create test run for given component
+ allow to run tests by component
3. meaning:
+ why
+ functional tests do not depend on localization
+ the list of language dependent tests might be
different for each language
+ feature tests are not well described; are done only
once by one or few dedicated people
+ requirements:
+ function test appears only once in test run; can be
approved by any user with any locale and any platform
+ language dependent tests must be approved more times
in all locales
+ test results must clearly show what tests were
done; functional tests are important by priority;
l10n tests are important by priority and language;
e.g. German is more important than Czech because
there are more active users
4. platform:
+ why:
+ some tests might depend on platform; for example the
open/save dialogs look different way on Linux,Windows,
and MAX; also KDE vs. GNOME
+ main requirements:
+ these tests must be done on all affected platforms
+ test results must clearly show where it was tested
5. Maintenance:
+ problems:
+ searching for a particular test case
+ put test cases into the right position
+ comparing en-US template with localized test cases
+ localizing the whole bunch of test cases into new
language
+ requirements:
+ balance the number of branches, testgroups,
subgroups, and test cases in subgroup
+ must be easy to copy test cases for new language
+ often switched criteria must be in high level;
for example, do people switch more often between
priority or category or language when they work
with the test cases?
6. Branch for new release:
+ requirements:
+ must be easy to do the branch
+ must be easy to see all tests for the given release
7. Test run creation:
+ requirements:
+ 1 or very small number of active test runs because it
is compilcated to switch between them and analyze
results from many test runs
+ create test run for a given priority (basic vs. full
test)
+ create test run for a given component (only Writer
modified)
8. Doing tests:
+ requirements:
+ must be easy to search what needs testing
+ must be possible to sort testing by priority and
component
+ priority is more important than component
because it does not help to have perfectly tested
Writer when Base is not tested at all
9. Test analyze:
+ requirements:
+ must be easy to see what was tested by priority,
component, and language
+ less important language tests should not fuzz
test results by priority. e.g. it would be nice to see
that 100% of P1 tests are finished on en-US, de, fr,
it, and ja.
----------------------------------
Grrr, are you lost? I am a bit. What to do now? :-)
I still do not see the best solution. Let's compare some stuff by
importance:
a) priority is more important than component
+ we do not have time to organize a test run because of a single
fix
=> most of the time we will test more fixes => all components
b) platform is not much important
+ there will be only few platform dependent test cases
+ I am not aware about any test that would be platform
and language dependent
=> there will only platform dependent functional tests
We could mention the requirement in the summary or test
description.
e) locale specific tests are questionable
+ we might limit the test cases to certain locales in summary or
test description
c) doing test is the most important activity because it will be done by
many people most of the time
d) test results analyze must be easy: because we want to often check
where we are
e) creating test run must take only few minutes
f) it must be reasonable to maintain localized test cases; it would be a
nightmare otherwise
g) the balance of the number of branches, testgroups, subgroups,
testcases in a group is important; long lists are always hard to cope
with
---------------------------------------
Grrr, I am even more lost now. How are you? Maybe, I should sleep to
sort the ideas and see a better picture :-)
Ok, let's try it from another point of view. There are not that many
reasonable solutions after all.
i) Product is "LibreOffice"
ii) Priority must be only in "Branch" or "TestGroup" because we want to
create test run by priority.
Branch is ugly because it would force many test runs => makes more
actions complicated
=> Priority must by in "TestGroup" name
iii) We do not want "Component in "Branch" name because it would create
too many test runs.
"Component" is not super important.
=> "Component" can be in "Testgroup", "Subgroup" or "Test case
summary" or nowhere.
iv) We want to clearly split language-dependent,
language-independent and feature tests. Because they need different
approach when creating and testing.
We do not want "Language" in "Branch name" because we do not want
100 branches for different languages. It would be nightmare to
create test run, do testing, analyze it.
We do not want Language in "Test case summary" because it would be
hard to search "en-US" test cases inside the flood of other test
cases.
=> Language must be in "TestGroup" or "Subgroup" name
v) "Platform" affects only few tescases
=> mention platform only in test case name
---------------------------------------------------------------------------
So, where are we? Still in a tunel or do we see a light?
ok, let's do some counting:
I) Criteria has variants:
+ priority => 4 variants
+ category => 7 variants
+ meaning => 3 variants
+ language => 100 variants
+ platform => 3 variants
II) Criteria and number of test cases:
The expected number of test cases is in brackets.
+ priority: P1(10), P2(50), P3(150), P4(?)
+ category: General(100), Writer(100), Calc(100),Math(20)
+ meaning: function(500), l10n(20 per language), feature(100)
+ platform: Linux(10), MAC(10), Windows(10)
III) Litmus levels and numbers:
We want the following numbers in Litmus:
+ product: 1
+ branch: 1-5 per LO version
+ testgroup: 3-20 (100 per language would be a nightmare)
+ subgroup: 3-20 (max 100 if we want to put language there)
---------------------------------------------------
I think that we come somewhere. I am sure that we will need a
compromise:
A. 3 branches, 4 test groups, 7/100 subgroups:
==============================================
+ LibreOffice product (three branches per release):
+ master function test branch (4 testgroups):
+ P1 testgroup (7 subgroups)
+ General
+ Base
+ Calc
+ Draw
+ Impress
+ Math
+ Writer
+ P2 testgroup (7 subgroups)
+ General
+ Base
+ ...
+ P3 ...
+ P4 ...
+ master l10n test branch (4 testgroups)
+ P1 test group (100 subgroups)
+ en
+ de
+ fr
+ ...
+ P2 test group (100 subgroups)
+ en
+ ...
+ P3 ...
+ P4 ...
+ master feature test
+ ???
+ 3.5 function test branch
+ 3.5 l10n ...
+ 3.5 feature ...
+ 3.6 function ...
+ ..
+ ...
B. 1 branch, 9-12 test groups, 7/100 subgroups
==============================================
LibreOffice product (one branch per release):
+ master branch (9-12 test groups):
+ P1 - function testgroup (7 subgroups)
+ General
+ Base
+ Calc
+ Draw
+ Impress
+ Math
+ Writer
+ P1 - l10n testgroup (100 subgroups)
+ en
+ de
+ fr
+ ...
+ P2 - function tesgroup(7 subgroups)
+ General
+ Base
+ ...
+ P2 - l10n testgroup (100 subgroups)
+ en
+ de
+ P3 - function tesgroup(7 subgroups)
+ P3 - l10n testgroup (100 subgroups)
+ P4 - function tesgroup(7 subgroups)
+ P4 - l10n testgroup (100 subgroups)
+ P4 - feature tesgroup (7 subgroups) (only P4?)
+ General
+ Base
+ Calc
+ 3.5 branch (9-12 test groups)
+ P1 - function
+ P1 - l10n
+ ...
+ 3.6 branch (9-12 test groups)
+ ...
C. 1 branch, 28 test groups, 101 subgroups
==========================================
+ LibreOffice product (one branch per release)
+ master branch (28 testgroups)
+ P1 - General (101 subgroups)
+ function tests
+ l10n tests - en
+ l10n tests - de
+ l10n tests - fr
+ ...
+ P1 - Base(101 subgroups)
+ function tests
+ l10n tests - en
+ l10n tests - de
+ ...
+ P1 - Calc(101 subgroups)
+ ...
+ P2 -General(101 subgroups)
+ P2 - Base(101 subgroups)
+ ...
+ 3.5 branch (28 testgoups)
+ 3.5 branch (28 testgroups)
+ ...
----------------------------------------------------------------------------
Do you see any other reasonable variant.
Anyway,I prefer the variant B because:
+ A does not have any real advantage, it just forces more test
runs and makes things more complicated
+ B is a nice compromise for any action
+ C does not split well function and l10n tests. It has too many
testgroups, so it would be a nightmare to check test results
and localize the l10n tests. The only advantage is that it
allows to create test run for Writer but I am not sure it if
is worth the effort
-----------------------------------------------------------------------------
Sigh, one more important thing. We might need to create the feature
branch earlier. Features are different for every release. We do not want
them in the master branch. We want them only in the release branch. We
want to branch the regression tests later from the master branch,
though.
I suggest to create a variant of B.
D. 2 branches, 8 testgroups, 7/100 subgroups:
+ LibreOffice product (two branches per release):
+ master branch (8 test groups):
+ P1 - function testgroup (7 subgroups)
+ General
+ Base
+ Calc
+ Draw
+ Impress
+ Math
+ Writer
+ P1 - l10n testgroup (100 subgroups)
+ en
+ de
+ fr
+ ...
+ P2 - function tesgroup(7 subgroups)
+ General
+ Base
+ ...
+ P2 - l10n testgroup (100 subgroups)
+ en
+ de
+ P3 - function tesgroup(7 subgroups)
+ P3 - l10n testgroup (100 subgroups)
+ P4 - function tesgroup(7 subgroups)
+ P4 - l10n testgroup (100 subgroups)
+ 3.5 regression branch (9-12 test groups)
+ P1 - function
+ P1 - l10n
+ ...
+ 3.5 feature test branch
+ ???
+ 3.6 regression branch (8 testgroups)
+ 3.6 feature test branch
+ ...
Does anyone reach this point?
Did I miss something any interesting criteria or condition?
Is the preferred solution really good in all actions?
Could we solve the localization stuff a better way? How?
Do you have a power to tell me what you think about it? :-)
Best Regards,
Petr
More information about the Libreoffice-qa
mailing list