[Libreoffice] Litmus testcase structure improvement.

[Yifan Jiang]

Hi all,

Referring to the attached discussion thread initialed by Jean-Baptiste (great
thanks!), we are trying to make testcase management tool Litmus
(https://tcm.documentfoundation.org/run_tests.cgi) better managed and
organized. The following explained all current states. As usual, comments are
welcome! Thanks for all the support!

[For QA people running cases]

    The run created by new method for 3.4.1:

        https://tcm.documentfoundation.org/run_tests.cgi?test_run_id=13

    Basically no practical changes from before. Meanwhile please
    notice:

    a. make sure the build version you are using is suitable for
    the corresponding test run. For example, for the test run

        - LibreOffice 3.4.0 RC Regression test

    libreoffice build 3.4.0.x is needed.

    b. Please make sure the platform, operating system and build
    id information, which should be filled before the testing,
    exactly reflecting the testing environment.

    c. Attaching a bug id to a failed case result would be very
    nice!

[For Litmus admins]

    We are trying to make every thing much more clear and easy to
    update. The current test case organization changes:

    a. Divide test case set to Regression test and New Feature
    test ([1])

    b. Regression Test

        A branch *Master Regression TC* was created
        specifically. So that we can all work on a consistent
        test case base, from which new branches can be always
        *cloned*.

        Keep all new/updated cases in this branch would be
        necessary to make everything inside reusable.
        
        That is to say, we use the Master branch to keep a
        reusable and continuously maintanable test cases storage,
        but NOT for creating test runs.

        When a new build (big version as x.x) will come, we

            1. clone (Recursively) a new regression test branch
               for the build from Master branch, like

               3.4 Regresstion Test (branch)
               3.5 Regresstion Test (branch)

               ...
                
            2. through the branches, create a test run for the corresponding
            builds:

               libreoffice 3.4.0 rc regresstion test (run created from 3.4 branch)
               libreoffice 3.4.1 rc regresstion test (run created from 3.4 branch)
               libreoffice 3.4.2 rc regresstion test (run created from 3.4 branch)

               libreoffice 3.5.0 rc regresstion test (run created from 3.5 branch)
               libreoffice 3.5.1 rc regresstion test (run created from 3.5 branch)
               libreoffice 3.5.2 rc regresstion test (run created from 3.5 branch)

                ...

            *NOTICE*
            
              If some new cases needed, please create them in the
              Master Testcase branch. Thus we can use the new
              regression test case set next time creating
              branches([2]).

              I have noticed a new set of pt-BR test cases
              created in '3.4 Regression test' branch today, I will
              'sync' them to the Master branch later.

    c. New Feature Test

        A branch *Master Feature TC* was created
        specifically. This is more like an outline without cases
        (but with group subgroup info) filled in. So for each new
        big version (x.x),

            1. clone from the Master Feature branch:

                3.4 Feature Test (branch)
                
                3.5 Feature Test (branch)
                
                3.6 Feature Test (branch)

            2. we create a single run for each big version:

                libreoffice 3.4 feature test (run created from 3.4 branch)

                libreoffice 3.5 feature test (run created from 3.5 branch)

                libreoffice 3.6 feature test (run created from 3.6 branch)

                ...

            3. we 'seek' and write new feature cases in the
              corresponding subgroup according to git log,
              mailing list, release notes or so.

            4. we update important feature case to Master
               Regression TC branch

[Footnotes]

    [1] Regression and Feature test are there for quite different
    purpose:

        The Regression test would hold relatively stable set of
        test cases, which should be ran in each of the release
        build.

        The New Feature test would be dynamically changed based
        on a big version update (3.3->3.4). Meanwhile New feature
        test might be run from even early development build
        phase.

    [2] As you may notice, by this process we didn't really
        change the running branch.  However I didn't find an easy way
        to sync test cases between branches :( A workaround method to
        'link' new test cases is:

        1. Add a new test cases in the Master branch, and
        assign the cases to a proper subgroup

        2. Edit current running branch's subgroup to link a new
        case from the Master branch to the current running
        one. (manage subgroups -> select Product as libreoffice
        and Banch as currently running -> edit subgroups ->
        manage testcases for this subgroup)

        Please also let me know if things could be better.

-- 
  Yifan Jiang
  Libreoffice
  Contact: yifan - irc.freenode.net/libreoffice
  =============================================  
  http://www.libreoffice.org/
  http://www.documentfoundation.org/

-------------- next part --------------
An embedded message was scrubbed...
From: Yifan Jiang <yfjiang at novell.com>
Subject: Re: [Libreoffice] QA-hints ?   ->  question for devs !
Date: Tue, 14 Jun 2011 14:58:44 +0800
Size: 6320
URL: <http://lists.freedesktop.org/archives/libreoffice/attachments/20110622/3bdc260b/attachment.mht>