[Libreoffice-qa] Test case naming

[Rimas Kudelis]

Hi all,

I took a quick look at how testruns are created and what we can and
cannot choose when creating a testrun. My suggestion is based on the
fact that each testrun comprises of

* all subgroups in
* one or more testgroups in
* exactly one branch in
* exactly one product.

That is, a testrun cannot span through multiple branches or products,
nor can you specify which subgroups to add to it. You can only choose
groups.

First of all, my suggestion is to merge the three branches per product
version that we currently have into one (which is I think what the word
Branch refers to in this context). Thus we would have the master branch,
3.4 branch, 3.5 branch etc.

More details below and in the attachment.

2011.11.15 07:34, Yifan Jiang rašė:
> On Mon, Nov 14, 2011 at 10:25:28PM +0200, Rimas Kudelis wrote:
>
>>>> That is to say, even before we sort out how order of the test cases
>>>> could be implemented, we can always create specific test runs on
>>>> demand via the information of the priority "tags".
>>> BTW: How do you suggest to create the priority "tag"? Is there any
>>> better solution than to put it into prefix of the test case summary?
>> Well, as an alternative, branches/groups/subgroups could be reviewed
>> again. :)
> The idea is actually brilliant! I also thought a bit in this way, but didn't
> conclude anything cenrtain at the moment. The problems I can think by
> leveraging branch/group/subgroup is
>
>     1. The test cases priority heavily depend on Litmus tool that it doesn't
>     carry the information with their summaries. Instead, the group/subgroup/branch
>     where they are in tell us the priority.
>
>     2. The maintainence effort and process may be increased a bit, but in a
>     acceptable amount (at least for me).
>
> The benifits are:
>
>     1. Creating test runs can be easier that higher level definition would
>     allow admins to create a specific priority test run simply by selecting
>     corresponding subgroup, group or branch of test cases, instead of reviwing
>     each test case's title.
>
>     2. The maintenence of test cases is assumed to be more elegant and
>     intuitive. For example, we can change a test case's priority by change its
>     corresponding subgroup, group or branch, instead of changing test case's
>     title.
>
> The following is something they would look like, assume we don't change a lot
> the whole structure of current organization.
>
> Overall satistics:
>
>     We now have 3 branches:
>
>         master regression branch
>             - 1 group
>                 - 7 subgroups in each group
>         master l10n branch
>             - 4 groups
>                 - 7 subgroups in each group
>         master feature branch
>             - we may not consider case priority here?

Current scheme does not allow creating a catch-all testrun comprising
all tests for a single product/branch. I don't know whether or not such
ability is desired, but it looks logical to me. Also, maybe I shouldn't
look that far into future, but I hope that there will come a day when
proper localization of testcases will be possible (that is, instead of
creating a clone of testcase X in another language, we would actually be
able to translate testcase X into that language). With that in mind,
current testgroups (which represent different locales) would become
unneeded.

> Case 1. Carry priority information with subgroups name
>
>     Prototype:
>
>         Branch Master Function Regression Test
>
>         Group: Function test
>
>             Subgroup p1 writer
>             Subgroup p2 writer
>             Subgroup p3 writer
>             Subgroup p4 writer            
>             Subgroup p1 calc
>             Subgroup p2 calc
>             Subgroup p3 calc
>             ...

This scheme would make it impossible to create a testrun for e.g. just
P1 tests. Or just writer tests. I guess this is pretty much the same as
what we have now, so all criticism from above paragraph applies here too.

> Case 2. Carry priority information with groups name
>
>     Prototype:
>
>         Branch Master Function Regression Test
>
>         Group: Function test p1
>             Subgroup writer
>             Subgroup calc
>             ...        
>         Group: Function test p2
>             Subgroup writer
>             Subgroup calc
>             ...        
>         Group: Function test p3
>             Subgroup writer
>             Subgroup calc
>             ...        

This is probably OK if you don't mind having to create three testruns
for each version and be unable to create testruns for separate
components. I think you forgot locales here though. I assume that for
each locale/priority there would be a separate testgroup?

> Case 3. Carry priority information with branches
>
>     Branch Master Function Regression Test p1
>     Branch Master Function Regression Test p2
>     Branch Master Function Regression Test p3
>     Branch Master Function Regression Test p4
>         All the structure inside different priority branches are the same.
>
>     Branch Master L10n Regression Test p1
>     Branch Master L10n Regression Test p2
>     Branch Master L10n Regression Test p3
>     Branch Master L10n Regression Test p4
>         All the structure inside different priority branches are the same.
>
> I am slightly inclined to the Case 3, it still keeps most of original things
> (groups and subroups), which have already been divided in a clear logic and
> not that easy to get familiar with. Also Case 3 is the most easy way if we
> want to create different runs in different phase of release.

IMO, Case 3 is the worst case, because for each version of LibreOffice,
you would have to create 4*3=12 testruns – one for each priority/test
type combination.


> Intuitively, we can "feel" all the three cases by describe a particular l10n
> test case in each:
>
> Case 1 - subgroup.
>
>     This is a P1 writer test case in English locale for regression testing
>
> Case 2 - group
>
>     This is a P1 English locale writer test case for regression testing
>
> Case 3 - branch
>
>     This is a P1 regression test case for writer in an English locale
>
> I feel Case 2 is hard to describe :) Case 1 is okay, but it doesn't really
> take advantage against Case 3. 
>
> It would be interesting if you share some your ideas of cons/prons of them. :)

My suggestion is to have a single branch, but carry Priority, Locale and
Component information in the Testgroup, and to represent test type by a
subgroup. That is, what is now a branch, would become a subgroup, and
everything else would become a testgroup (see the attached PDF file).
This allows to:
* create testruns based on priority, locale, component, or any
combination of these
* create a single catch-all testrun for a single version of LibO
* share the same General Functional Tests subgroup between testgroups
designated for different locales (that is, you would create this
subgroup once, and add it to all locale groups)
* drop 75% of the testgroups (there would be about 28 of them initially)
if/when proper testcase localization is implemented, still not rendering
the remaining testgroups useless.


Well, that's probably about it. However, as people who actually
coordinate and perform testing, you may have a better clue than me, so
you may have a reason not to buy what I'm selling here. :)

Regards,
Rimas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Litmus reorg.pdf
Type: application/pdf
Size: 41235 bytes
Desc: not available
URL: <http://lists.freedesktop.org/archives/libreoffice-qa/attachments/20111115/f49a9e42/attachment-0001.pdf>