[Libreoffice-qa] Test case naming

[Petr Mladek]

Hi guys,

first, I am sorry that this mail does react on the last mail from
Rimas. I started to write this replay in the morning. I had to
interrupt it for some reasons. I think that the ideas mentioned below
still make sense. Though, I might change my preference after reading
Rimas's reply.


Yifan Jiang píše v Út 15. 11. 2011 v 13:34 +0800:
> On Mon, Nov 14, 2011 at 10:25:28PM +0200, Rimas Kudelis wrote:
> > Well, as an alternative, branches/groups/subgroups could be reviewed
> > again. :)
> 
> The idea is actually brilliant!

Fully agree!

> The following is something they would look like, assume we don't change a lot
> the whole structure of current organization.
> 
> Case 1. Carry priority information with subgroups name

> Case 2. Carry priority information with groups name

> Case 3. Carry priority information with branches

> I am slightly inclined to the Case 3, it still keeps most of original things
> (groups and subroups), which have already been divided in a clear logic and
> not that easy to get familiar with. Also Case 3 is the most easy way if we
> want to create different runs in different phase of release.

I think that creating test run is only one scenario. I think that we
need to think about all use cases:

A. Maintenance
--------------

The are few actions:

	+ localize test - has two subactions:

		+ creating the structure:

	                If we could copy the whole group,
                        solution 1 is fastest because there
                        are less groups and branches. Solution 2
                        is good because you need not switch between
                        branches.

                        If we need to create the whole
                        structure from scratch, it does not
                        matter because the total number of
                        subgroups is always the same


		+ checking differences and localizing

                        If you select another branch in the search dialog,
                        you need to select group and subgroup again

			=> Solution 1 is better than 2
                        and 2 is better than 3 because
                        if is easier to switch between
                        subgroups; it is easier to between
                        groups than branches


	+ adding/updating test cases:

		+ IMHO, it is similar to checking and localizing =>
                  1 is better than 2 and 2 is better than 3
                  because it is easier to switch subgroups than groups
                  and switch groups than branches
           

B. Creating Branches for new release
------------------------------------

	+ 1 or 2 are easier than 3 because they have less branches


C. Creating Test Runs
---------------------

	+ I do not know how this works. You say that 3 is the easiest
          way because you just create run from a branch?

	+ IMHO, 1 would be nightmare because you would need
          to select to many subgroups manually if you want to test only
          P1 test.


D. Testing
----------

This is very important because many people will do this many time!!!

I think that we have 2 actions:	

	+ doing the test

		People need to find easily what tests are not finished.
                You need to enter a test run to see the statistic

                => we should have as few active test runs as possible

                if 3 produces more test runs, I am against it

                => 1 or 2 is better

	+ analyzing results:

		I am not sure how we check the state of testing; the best solution
                would be to check on page and see how many tests are finished;
                how many tests succeded and failed

		I guess that the solution 3 is bad from this point of view.


Summary:
--------

3 is bad for testing. 1 is bad for creating test runs. 2 is the best
compromise in most situations.

=> I prefer 2.

Note that 2 has better granularity. We will have many branches in the
long term for released products. Solution 1 produces too many subgroups
in a single group.


=================


Hmm, when I think about it. I am somehow afraid that we would have too
many subgroups in the l10n branch. It does not mater how we split it
but it will be hard to maintain.

Do we really expect that many l10n-specific test cases for Draw,
Impress, Base, or Calc? Do we really need the subgroups?

What about the following structure:

+ function tests:

	+ P1
		+ General
		+ Base
		+ Calc
                + ...

        + P2

		+ General
		+ Base
		+ ...

+ l10n test:

	+ P1
		+ en
		+ de
		+ fr
		+ it

	+ P2
		+ en
		+ de
		+ fr
		+ it
	


> Intuitively, we can "feel" all the three cases by describe a particular l10n
> test case in each:
> 
> Case 1 - subgroup.
> 
>     This is a P1 writer test case in English locale for regression testing
> 
> Case 2 - group
> 
>     This is a P1 English locale writer test case for regression testing
> 
> Case 3 - branch
> 
>     This is a P1 regression test case for writer in an English locale
> 
> I feel Case 2 is hard to describe :) Case 1 is okay, but it doesn't really
> take advantage against Case 3. 

I do not see any big differences in how we describe the structure. We
split test cases according to several criteria on different levels. We
just change the mapping of the criteria and the level.


Best Regards,
Petr