[Libreoffice-qa] Test Structure in Litmus
Yifan Jiang
yfjiang at suse.com
Wed Nov 16 00:38:54 PST 2011
Hi Petr,
Thanks for the detailed summary, I hope I can review this before sending
mine:) If I understand you correctly, scheme D ignores a requirement, doesn't
it?
> 7. Test run creation:
>
> + requirements:
> + create test run for a given component (only Writer
> modified)
In the latest mail I just replied to "Test case naming" I shared a similar
idea. I can see the main differences of ours focused on 2 points:
1. Do we need components split in L10n tests?
I have ever considered the leraning curve when l10n and function test
doesn't have a consistent scheme. But from reconsidering, it is probably
natural for a user to see such language split when clicking a
Localizatoin test group. Plus it reduces the test group size
significantly, which is my favorite as well. So I give up this concern :)
Another question is still about how many cases will we have. One thing I
didn't mention is with this scheme, when we create a test case we need
to review all the cases to make sure they were not created
before. Currently we only have 23 cases, it is okay if we review all of
the cases summaries when creating a new one. But if we get even more cases (> 50
for example), it becomes painful and hard to manage.
2. Do we need an extra feature branch?
It is pretty fine for me to have another branch, I did actually
prefer to leverage branches in a reasonable scale :)
Best wishes,
Yifan
On Tue, Nov 15, 2011 at 07:44:59PM +0100, Petr Mladek wrote:
> Hi,
>
> I am going to summarize the ideas from the mail "Test case naming".
>
> I am sorry for the long mail. The problem is not easy. There are many
> ideas. I tried to sort them from different point of view. I hope that I
> did not miss anything important and it will help to find the final
> conclusion.
>
> ----------------------------------------------------------
>
> Let's start by a short summary.
>
> Litmus allows to split test cases by:
>
> + product
> + branch
> + testgroup
> + subgroup
>
>
> We want to split test cases by:
>
> + priority (P1,P2,P3,P4)
> + component (General,Base,Calc,Draw,Impress,Math,Writer)
> + meaning (functional, l10n, feature)
> + l10n tests in languages (en,de,fr,it,ja,.............)
> + platform (Linux,Windows,MAC)
>
> We need to take care of the following scenarios:
>
> + maintenance:
> + creating test cases
> + updating test cases
> + localization
> + branch for new release
> + test run for new release/build
> + doing tests
> + analyzing results
>
> -----------------------------------
>
> Let's look at some conditions:
>
> 1. priority:
>
> + why
> + reduced test run for bugfix releases
> + overal prioritization of the work
>
> + main requirements:
>
> + allow to create test run for given priority
> + easily run tests according to priority
> + easily see test results by priority
>
>
> 2. component:
>
> + why
>
> + reduced test run if only writer is modified
> + helps to split work by expertize
>
> + main requirements:
>
> + allow to create test run for given component
> + allow to run tests by component
>
> 3. meaning:
>
> + why
>
> + functional tests do not depend on localization
> + the list of language dependent tests might be
> different for each language
> + feature tests are not well described; are done only
> once by one or few dedicated people
>
> + requirements:
>
> + function test appears only once in test run; can be
> approved by any user with any locale and any platform
> + language dependent tests must be approved more times
> in all locales
> + test results must clearly show what tests were
> done; functional tests are important by priority;
> l10n tests are important by priority and language;
> e.g. German is more important than Czech because
> there are more active users
>
>
> 4. platform:
>
> + why:
>
> + some tests might depend on platform; for example the
> open/save dialogs look different way on Linux,Windows,
> and MAX; also KDE vs. GNOME
>
> + main requirements:
>
> + these tests must be done on all affected platforms
> + test results must clearly show where it was tested
>
>
> 5. Maintenance:
>
> + problems:
>
> + searching for a particular test case
> + put test cases into the right position
> + comparing en-US template with localized test cases
> + localizing the whole bunch of test cases into new
> language
>
> + requirements:
>
> + balance the number of branches, testgroups,
> subgroups, and test cases in subgroup
> + must be easy to copy test cases for new language
> + often switched criteria must be in high level;
> for example, do people switch more often between
> priority or category or language when they work
> with the test cases?
>
>
> 6. Branch for new release:
>
> + requirements:
>
> + must be easy to do the branch
> + must be easy to see all tests for the given release
>
>
> 7. Test run creation:
>
> + requirements:
>
> + 1 or very small number of active test runs because it
> is compilcated to switch between them and analyze
> results from many test runs
> + create test run for a given priority (basic vs. full
> test)
> + create test run for a given component (only Writer
> modified)
>
> 8. Doing tests:
>
> + requirements:
>
> + must be easy to search what needs testing
> + must be possible to sort testing by priority and
> component
> + priority is more important than component
> because it does not help to have perfectly tested
> Writer when Base is not tested at all
>
> 9. Test analyze:
>
> + requirements:
>
> + must be easy to see what was tested by priority,
> component, and language
> + less important language tests should not fuzz
> test results by priority. e.g. it would be nice to see
> that 100% of P1 tests are finished on en-US, de, fr,
> it, and ja.
>
>
> ----------------------------------
>
> Grrr, are you lost? I am a bit. What to do now? :-)
>
> I still do not see the best solution. Let's compare some stuff by
> importance:
>
> a) priority is more important than component
>
> + we do not have time to organize a test run because of a single
> fix
> => most of the time we will test more fixes => all components
>
> b) platform is not much important
>
> + there will be only few platform dependent test cases
> + I am not aware about any test that would be platform
> and language dependent
> => there will only platform dependent functional tests
>
> We could mention the requirement in the summary or test
> description.
>
> e) locale specific tests are questionable
>
> + we might limit the test cases to certain locales in summary or
> test description
>
> c) doing test is the most important activity because it will be done by
> many people most of the time
>
> d) test results analyze must be easy: because we want to often check
> where we are
>
> e) creating test run must take only few minutes
>
> f) it must be reasonable to maintain localized test cases; it would be a
> nightmare otherwise
>
> g) the balance of the number of branches, testgroups, subgroups,
> testcases in a group is important; long lists are always hard to cope
> with
>
>
> ---------------------------------------
>
>
> Grrr, I am even more lost now. How are you? Maybe, I should sleep to
> sort the ideas and see a better picture :-)
>
> Ok, let's try it from another point of view. There are not that many
> reasonable solutions after all.
>
>
> i) Product is "LibreOffice"
>
> ii) Priority must be only in "Branch" or "TestGroup" because we want to
> create test run by priority.
>
> Branch is ugly because it would force many test runs => makes more
> actions complicated
>
> => Priority must by in "TestGroup" name
>
>
> iii) We do not want "Component in "Branch" name because it would create
> too many test runs.
>
> "Component" is not super important.
>
> => "Component" can be in "Testgroup", "Subgroup" or "Test case
> summary" or nowhere.
>
>
> iv) We want to clearly split language-dependent,
> language-independent and feature tests. Because they need different
> approach when creating and testing.
>
> We do not want "Language" in "Branch name" because we do not want
> 100 branches for different languages. It would be nightmare to
> create test run, do testing, analyze it.
>
> We do not want Language in "Test case summary" because it would be
> hard to search "en-US" test cases inside the flood of other test
> cases.
>
> => Language must be in "TestGroup" or "Subgroup" name
>
> v) "Platform" affects only few tescases
>
> => mention platform only in test case name
>
>
>
> ---------------------------------------------------------------------------
>
>
> So, where are we? Still in a tunel or do we see a light?
>
> ok, let's do some counting:
>
> I) Criteria has variants:
>
> + priority => 4 variants
> + category => 7 variants
> + meaning => 3 variants
> + language => 100 variants
> + platform => 3 variants
>
>
> II) Criteria and number of test cases:
>
> The expected number of test cases is in brackets.
>
> + priority: P1(10), P2(50), P3(150), P4(?)
> + category: General(100), Writer(100), Calc(100),Math(20)
> + meaning: function(500), l10n(20 per language), feature(100)
> + platform: Linux(10), MAC(10), Windows(10)
>
>
> III) Litmus levels and numbers:
>
> We want the following numbers in Litmus:
>
> + product: 1
> + branch: 1-5 per LO version
> + testgroup: 3-20 (100 per language would be a nightmare)
> + subgroup: 3-20 (max 100 if we want to put language there)
>
>
> ---------------------------------------------------
>
> I think that we come somewhere. I am sure that we will need a
> compromise:
>
>
> A. 3 branches, 4 test groups, 7/100 subgroups:
> ==============================================
>
> + LibreOffice product (three branches per release):
>
> + master function test branch (4 testgroups):
> + P1 testgroup (7 subgroups)
> + General
> + Base
> + Calc
> + Draw
> + Impress
> + Math
> + Writer
> + P2 testgroup (7 subgroups)
> + General
> + Base
> + ...
> + P3 ...
> + P4 ...
>
> + master l10n test branch (4 testgroups)
> + P1 test group (100 subgroups)
> + en
> + de
> + fr
> + ...
> + P2 test group (100 subgroups)
> + en
> + ...
> + P3 ...
> + P4 ...
> + master feature test
> + ???
>
> + 3.5 function test branch
> + 3.5 l10n ...
> + 3.5 feature ...
> + 3.6 function ...
> + ..
>
> + ...
>
>
> B. 1 branch, 9-12 test groups, 7/100 subgroups
> ==============================================
>
> LibreOffice product (one branch per release):
>
> + master branch (9-12 test groups):
> + P1 - function testgroup (7 subgroups)
> + General
> + Base
> + Calc
> + Draw
> + Impress
> + Math
> + Writer
> + P1 - l10n testgroup (100 subgroups)
> + en
> + de
> + fr
> + ...
> + P2 - function tesgroup(7 subgroups)
> + General
> + Base
> + ...
> + P2 - l10n testgroup (100 subgroups)
> + en
> + de
> + P3 - function tesgroup(7 subgroups)
> + P3 - l10n testgroup (100 subgroups)
> + P4 - function tesgroup(7 subgroups)
> + P4 - l10n testgroup (100 subgroups)
> + P4 - feature tesgroup (7 subgroups) (only P4?)
> + General
> + Base
> + Calc
>
>
> + 3.5 branch (9-12 test groups)
> + P1 - function
> + P1 - l10n
> + ...
> + 3.6 branch (9-12 test groups)
> + ...
>
>
>
> C. 1 branch, 28 test groups, 101 subgroups
> ==========================================
>
> + LibreOffice product (one branch per release)
>
> + master branch (28 testgroups)
> + P1 - General (101 subgroups)
> + function tests
> + l10n tests - en
> + l10n tests - de
> + l10n tests - fr
> + ...
> + P1 - Base(101 subgroups)
> + function tests
> + l10n tests - en
> + l10n tests - de
> + ...
> + P1 - Calc(101 subgroups)
> + ...
> + P2 -General(101 subgroups)
> + P2 - Base(101 subgroups)
> + ...
>
> + 3.5 branch (28 testgoups)
> + 3.5 branch (28 testgroups)
> + ...
>
>
> ----------------------------------------------------------------------------
>
> Do you see any other reasonable variant.
>
> Anyway,I prefer the variant B because:
>
> + A does not have any real advantage, it just forces more test
> runs and makes things more complicated
> + B is a nice compromise for any action
> + C does not split well function and l10n tests. It has too many
> testgroups, so it would be a nightmare to check test results
> and localize the l10n tests. The only advantage is that it
> allows to create test run for Writer but I am not sure it if
> is worth the effort
>
> -----------------------------------------------------------------------------
>
> Sigh, one more important thing. We might need to create the feature
> branch earlier. Features are different for every release. We do not want
> them in the master branch. We want them only in the release branch. We
> want to branch the regression tests later from the master branch,
> though.
>
> I suggest to create a variant of B.
>
>
> D. 2 branches, 8 testgroups, 7/100 subgroups:
>
> + LibreOffice product (two branches per release):
>
> + master branch (8 test groups):
> + P1 - function testgroup (7 subgroups)
> + General
> + Base
> + Calc
> + Draw
> + Impress
> + Math
> + Writer
> + P1 - l10n testgroup (100 subgroups)
> + en
> + de
> + fr
> + ...
> + P2 - function tesgroup(7 subgroups)
> + General
> + Base
> + ...
> + P2 - l10n testgroup (100 subgroups)
> + en
> + de
> + P3 - function tesgroup(7 subgroups)
> + P3 - l10n testgroup (100 subgroups)
> + P4 - function tesgroup(7 subgroups)
> + P4 - l10n testgroup (100 subgroups)
>
> + 3.5 regression branch (9-12 test groups)
> + P1 - function
> + P1 - l10n
> + ...
> + 3.5 feature test branch
> + ???
> + 3.6 regression branch (8 testgroups)
> + 3.6 feature test branch
> + ...
>
>
> Does anyone reach this point?
>
> Did I miss something any interesting criteria or condition?
> Is the preferred solution really good in all actions?
> Could we solve the localization stuff a better way? How?
>
> Do you have a power to tell me what you think about it? :-)
>
>
> Best Regards,
> Petr
>
More information about the Libreoffice-qa
mailing list