[Libreoffice-qa] Test case naming

[Yifan Jiang]

Hi Petr, Rimas and all,

On Mon, Nov 14, 2011 at 10:25:28PM +0200, Rimas Kudelis wrote:

> >> That is to say, even before we sort out how order of the test cases
> >> could be implemented, we can always create specific test runs on
> >> demand via the information of the priority "tags".
> > BTW: How do you suggest to create the priority "tag"? Is there any
> > better solution than to put it into prefix of the test case summary?
> 
> Well, as an alternative, branches/groups/subgroups could be reviewed
> again. :)

The idea is actually brilliant! I also thought a bit in this way, but didn't
conclude anything cenrtain at the moment. The problems I can think by
leveraging branch/group/subgroup is

    1. The test cases priority heavily depend on Litmus tool that it doesn't
    carry the information with their summaries. Instead, the group/subgroup/branch
    where they are in tell us the priority.

    2. The maintainence effort and process may be increased a bit, but in a
    acceptable amount (at least for me).

The benifits are:

    1. Creating test runs can be easier that higher level definition would
    allow admins to create a specific priority test run simply by selecting
    corresponding subgroup, group or branch of test cases, instead of reviwing
    each test case's title.

    2. The maintenence of test cases is assumed to be more elegant and
    intuitive. For example, we can change a test case's priority by change its
    corresponding subgroup, group or branch, instead of changing test case's
    title.

The following is something they would look like, assume we don't change a lot
the whole structure of current organization.

Overall satistics:

    We now have 3 branches:

        master regression branch
            - 1 group
                - 7 subgroups in each group
        master l10n branch
            - 4 groups
                - 7 subgroups in each group
        master feature branch
            - we may not consider case priority here?
        
Case 1. Carry priority information with subgroups name

    Prototype:

        Branch Master Function Regression Test

        Group: Function test

            Subgroup p1 writer
            Subgroup p2 writer
            Subgroup p3 writer
            Subgroup p4 writer            
            Subgroup p1 calc
            Subgroup p2 calc
            Subgroup p3 calc
            ...

Case 2. Carry priority information with groups name

    Prototype:

        Branch Master Function Regression Test

        Group: Function test p1
            Subgroup writer
            Subgroup calc
            ...        
        Group: Function test p2
            Subgroup writer
            Subgroup calc
            ...        
        Group: Function test p3
            Subgroup writer
            Subgroup calc
            ...        

Case 3. Carry priority information with branches

    Branch Master Function Regression Test p1
    Branch Master Function Regression Test p2
    Branch Master Function Regression Test p3
    Branch Master Function Regression Test p4
        All the structure inside different priority branches are the same.

    Branch Master L10n Regression Test p1
    Branch Master L10n Regression Test p2
    Branch Master L10n Regression Test p3
    Branch Master L10n Regression Test p4
        All the structure inside different priority branches are the same.

I am slightly inclined to the Case 3, it still keeps most of original things
(groups and subroups), which have already been divided in a clear logic and
not that easy to get familiar with. Also Case 3 is the most easy way if we
want to create different runs in different phase of release.

Intuitively, we can "feel" all the three cases by describe a particular l10n
test case in each:

Case 1 - subgroup.

    This is a P1 writer test case in English locale for regression testing

Case 2 - group

    This is a P1 English locale writer test case for regression testing

Case 3 - branch

    This is a P1 regression test case for writer in an English locale

I feel Case 2 is hard to describe :) Case 1 is okay, but it doesn't really
take advantage against Case 3. 

It would be interesting if you share some your ideas of cons/prons of them. :)

Best wishes,
Yifan