[Libreoffice-qa] Test Structure in Litmus
Yifan Jiang
yfjiang at suse.com
Tue Nov 15 23:38:01 PST 2011
Hi Petr,
Hehe, I missed this mail before I started summarizing today. I'll have a
review for sure :)
Best wishes,
Yifan
On Tue, Nov 15, 2011 at 07:44:59PM +0100, Petr Mladek wrote:
> Hi,
>
> I am going to summarize the ideas from the mail "Test case naming".
>
> I am sorry for the long mail. The problem is not easy. There are many
> ideas. I tried to sort them from different point of view. I hope that I
> did not miss anything important and it will help to find the final
> conclusion.
>
> ----------------------------------------------------------
>
> Let's start by a short summary.
>
> Litmus allows to split test cases by:
>
> + product
> + branch
> + testgroup
> + subgroup
>
>
> We want to split test cases by:
>
> + priority (P1,P2,P3,P4)
> + component (General,Base,Calc,Draw,Impress,Math,Writer)
> + meaning (functional, l10n, feature)
> + l10n tests in languages (en,de,fr,it,ja,.............)
> + platform (Linux,Windows,MAC)
>
> We need to take care of the following scenarios:
>
> + maintenance:
> + creating test cases
> + updating test cases
> + localization
> + branch for new release
> + test run for new release/build
> + doing tests
> + analyzing results
>
> -----------------------------------
>
> Let's look at some conditions:
>
> 1. priority:
>
> + why
> + reduced test run for bugfix releases
> + overal prioritization of the work
>
> + main requirements:
>
> + allow to create test run for given priority
> + easily run tests according to priority
> + easily see test results by priority
>
>
> 2. component:
>
> + why
>
> + reduced test run if only writer is modified
> + helps to split work by expertize
>
> + main requirements:
>
> + allow to create test run for given component
> + allow to run tests by component
>
> 3. meaning:
>
> + why
>
> + functional tests do not depend on localization
> + the list of language dependent tests might be
> different for each language
> + feature tests are not well described; are done only
> once by one or few dedicated people
>
> + requirements:
>
> + function test appears only once in test run; can be
> approved by any user with any locale and any platform
> + language dependent tests must be approved more times
> in all locales
> + test results must clearly show what tests were
> done; functional tests are important by priority;
> l10n tests are important by priority and language;
> e.g. German is more important than Czech because
> there are more active users
>
>
> 4. platform:
>
> + why:
>
> + some tests might depend on platform; for example the
> open/save dialogs look different way on Linux,Windows,
> and MAX; also KDE vs. GNOME
>
> + main requirements:
>
> + these tests must be done on all affected platforms
> + test results must clearly show where it was tested
>
>
> 5. Maintenance:
>
> + problems:
>
> + searching for a particular test case
> + put test cases into the right position
> + comparing en-US template with localized test cases
> + localizing the whole bunch of test cases into new
> language
>
> + requirements:
>
> + balance the number of branches, testgroups,
> subgroups, and test cases in subgroup
> + must be easy to copy test cases for new language
> + often switched criteria must be in high level;
> for example, do people switch more often between
> priority or category or language when they work
> with the test cases?
>
>
> 6. Branch for new release:
>
> + requirements:
>
> + must be easy to do the branch
> + must be easy to see all tests for the given release
>
>
> 7. Test run creation:
>
> + requirements:
>
> + 1 or very small number of active test runs because it
> is compilcated to switch between them and analyze
> results from many test runs
> + create test run for a given priority (basic vs. full
> test)
> + create test run for a given component (only Writer
> modified)
>
> 8. Doing tests:
>
> + requirements:
>
> + must be easy to search what needs testing
> + must be possible to sort testing by priority and
> component
> + priority is more important than component
> because it does not help to have perfectly tested
> Writer when Base is not tested at all
>
> 9. Test analyze:
>
> + requirements:
>
> + must be easy to see what was tested by priority,
> component, and language
> + less important language tests should not fuzz
> test results by priority. e.g. it would be nice to see
> that 100% of P1 tests are finished on en-US, de, fr,
> it, and ja.
>
>
> ----------------------------------
>
> Grrr, are you lost? I am a bit. What to do now? :-)
>
> I still do not see the best solution. Let's compare some stuff by
> importance:
>
> a) priority is more important than component
>
> + we do not have time to organize a test run because of a single
> fix
> => most of the time we will test more fixes => all components
>
> b) platform is not much important
>
> + there will be only few platform dependent test cases
> + I am not aware about any test that would be platform
> and language dependent
> => there will only platform dependent functional tests
>
> We could mention the requirement in the summary or test
> description.
>
> e) locale specific tests are questionable
>
> + we might limit the test cases to certain locales in summary or
> test description
>
> c) doing test is the most important activity because it will be done by
> many people most of the time
>
> d) test results analyze must be easy: because we want to often check
> where we are
>
> e) creating test run must take only few minutes
>
> f) it must be reasonable to maintain localized test cases; it would be a
> nightmare otherwise
>
> g) the balance of the number of branches, testgroups, subgroups,
> testcases in a group is important; long lists are always hard to cope
> with
>
>
> ---------------------------------------
>
>
> Grrr, I am even more lost now. How are you? Maybe, I should sleep to
> sort the ideas and see a better picture :-)
>
> Ok, let's try it from another point of view. There are not that many
> reasonable solutions after all.
>
>
> i) Product is "LibreOffice"
>
> ii) Priority must be only in "Branch" or "TestGroup" because we want to
> create test run by priority.
>
> Branch is ugly because it would force many test runs => makes more
> actions complicated
>
> => Priority must by in "TestGroup" name
>
>
> iii) We do not want "Component in "Branch" name because it would create
> too many test runs.
>
> "Component" is not super important.
>
> => "Component" can be in "Testgroup", "Subgroup" or "Test case
> summary" or nowhere.
>
>
> iv) We want to clearly split language-dependent,
> language-independent and feature tests. Because they need different
> approach when creating and testing.
>
> We do not want "Language" in "Branch name" because we do not want
> 100 branches for different languages. It would be nightmare to
> create test run, do testing, analyze it.
>
> We do not want Language in "Test case summary" because it would be
> hard to search "en-US" test cases inside the flood of other test
> cases.
>
> => Language must be in "TestGroup" or "Subgroup" name
>
> v) "Platform" affects only few tescases
>
> => mention platform only in test case name
>
>
>
> ---------------------------------------------------------------------------
>
>
> So, where are we? Still in a tunel or do we see a light?
>
> ok, let's do some counting:
>
> I) Criteria has variants:
>
> + priority => 4 variants
> + category => 7 variants
> + meaning => 3 variants
> + language => 100 variants
> + platform => 3 variants
>
>
> II) Criteria and number of test cases:
>
> The expected number of test cases is in brackets.
>
> + priority: P1(10), P2(50), P3(150), P4(?)
> + category: General(100), Writer(100), Calc(100),Math(20)
> + meaning: function(500), l10n(20 per language), feature(100)
> + platform: Linux(10), MAC(10), Windows(10)
>
>
> III) Litmus levels and numbers:
>
> We want the following numbers in Litmus:
>
> + product: 1
> + branch: 1-5 per LO version
> + testgroup: 3-20 (100 per language would be a nightmare)
> + subgroup: 3-20 (max 100 if we want to put language there)
>
>
> ---------------------------------------------------
>
> I think that we come somewhere. I am sure that we will need a
> compromise:
>
>
> A. 3 branches, 4 test groups, 7/100 subgroups:
> ==============================================
>
> + LibreOffice product (three branches per release):
>
> + master function test branch (4 testgroups):
> + P1 testgroup (7 subgroups)
> + General
> + Base
> + Calc
> + Draw
> + Impress
> + Math
> + Writer
> + P2 testgroup (7 subgroups)
> + General
> + Base
> + ...
> + P3 ...
> + P4 ...
>
> + master l10n test branch (4 testgroups)
> + P1 test group (100 subgroups)
> + en
> + de
> + fr
> + ...
> + P2 test group (100 subgroups)
> + en
> + ...
> + P3 ...
> + P4 ...
> + master feature test
> + ???
>
> + 3.5 function test branch
> + 3.5 l10n ...
> + 3.5 feature ...
> + 3.6 function ...
> + ..
>
> + ...
>
>
> B. 1 branch, 9-12 test groups, 7/100 subgroups
> ==============================================
>
> LibreOffice product (one branch per release):
>
> + master branch (9-12 test groups):
> + P1 - function testgroup (7 subgroups)
> + General
> + Base
> + Calc
> + Draw
> + Impress
> + Math
> + Writer
> + P1 - l10n testgroup (100 subgroups)
> + en
> + de
> + fr
> + ...
> + P2 - function tesgroup(7 subgroups)
> + General
> + Base
> + ...
> + P2 - l10n testgroup (100 subgroups)
> + en
> + de
> + P3 - function tesgroup(7 subgroups)
> + P3 - l10n testgroup (100 subgroups)
> + P4 - function tesgroup(7 subgroups)
> + P4 - l10n testgroup (100 subgroups)
> + P4 - feature tesgroup (7 subgroups) (only P4?)
> + General
> + Base
> + Calc
>
>
> + 3.5 branch (9-12 test groups)
> + P1 - function
> + P1 - l10n
> + ...
> + 3.6 branch (9-12 test groups)
> + ...
>
>
>
> C. 1 branch, 28 test groups, 101 subgroups
> ==========================================
>
> + LibreOffice product (one branch per release)
>
> + master branch (28 testgroups)
> + P1 - General (101 subgroups)
> + function tests
> + l10n tests - en
> + l10n tests - de
> + l10n tests - fr
> + ...
> + P1 - Base(101 subgroups)
> + function tests
> + l10n tests - en
> + l10n tests - de
> + ...
> + P1 - Calc(101 subgroups)
> + ...
> + P2 -General(101 subgroups)
> + P2 - Base(101 subgroups)
> + ...
>
> + 3.5 branch (28 testgoups)
> + 3.5 branch (28 testgroups)
> + ...
>
>
> ----------------------------------------------------------------------------
>
> Do you see any other reasonable variant.
>
> Anyway,I prefer the variant B because:
>
> + A does not have any real advantage, it just forces more test
> runs and makes things more complicated
> + B is a nice compromise for any action
> + C does not split well function and l10n tests. It has too many
> testgroups, so it would be a nightmare to check test results
> and localize the l10n tests. The only advantage is that it
> allows to create test run for Writer but I am not sure it if
> is worth the effort
>
> -----------------------------------------------------------------------------
>
> Sigh, one more important thing. We might need to create the feature
> branch earlier. Features are different for every release. We do not want
> them in the master branch. We want them only in the release branch. We
> want to branch the regression tests later from the master branch,
> though.
>
> I suggest to create a variant of B.
>
>
> D. 2 branches, 8 testgroups, 7/100 subgroups:
>
> + LibreOffice product (two branches per release):
>
> + master branch (8 test groups):
> + P1 - function testgroup (7 subgroups)
> + General
> + Base
> + Calc
> + Draw
> + Impress
> + Math
> + Writer
> + P1 - l10n testgroup (100 subgroups)
> + en
> + de
> + fr
> + ...
> + P2 - function tesgroup(7 subgroups)
> + General
> + Base
> + ...
> + P2 - l10n testgroup (100 subgroups)
> + en
> + de
> + P3 - function tesgroup(7 subgroups)
> + P3 - l10n testgroup (100 subgroups)
> + P4 - function tesgroup(7 subgroups)
> + P4 - l10n testgroup (100 subgroups)
>
> + 3.5 regression branch (9-12 test groups)
> + P1 - function
> + P1 - l10n
> + ...
> + 3.5 feature test branch
> + ???
> + 3.6 regression branch (8 testgroups)
> + 3.6 feature test branch
> + ...
>
>
> Does anyone reach this point?
>
> Did I miss something any interesting criteria or condition?
> Is the preferred solution really good in all actions?
> Could we solve the localization stuff a better way? How?
>
> Do you have a power to tell me what you think about it? :-)
>
>
> Best Regards,
> Petr
>
More information about the Libreoffice-qa
mailing list