[Libreoffice-qa] Test case naming
pmladek at suse.cz
Tue Nov 15 07:30:52 PST 2011
Rimas Kudelis píše v Út 15. 11. 2011 v 13:43 +0200:
> Hi all,
> I took a quick look at how testruns are created and what we can and
> cannot choose when creating a testrun. My suggestion is based on the
> fact that each testrun comprises of
> * all subgroups in
> * one or more testgroups in
> * exactly one branch in
> * exactly one product.
> That is, a testrun cannot span through multiple branches or products,
> nor can you specify which subgroups to add to it. You can only choose
This is important information. Thanks for looking at it.
> > Overall satistics:
> > We now have 3 branches:
> > master regression branch
> > - 1 group
> > - 7 subgroups in each group
> > master l10n branch
> > - 4 groups
> > - 7 subgroups in each group
> > master feature branch
> > - we may not consider case priority here?
> Current scheme does not allow creating a catch-all testrun comprising
> all tests for a single product/branch. I don't know whether or not such
> ability is desired, but it looks logical to me.
You are right. 1 run is the best solution. Well, 2 or three test runs
for a release still might be acceptable.
> Also, maybe I shouldn't look that far into future, but I hope that
> there will come a day when proper localization of testcases will be
> possible (that is, instead of creating a clone of testcase X in
> another language, we would actually be able to translate testcase X
> into that language). With that in mind, current testgroups (which
> represent different locales) would become unneeded.
I am not sure if the l10n tests will be pure word-to-word translation.
Some things are done completely different in different languages, for
example the layout of the letter template, right-to-left language
features, decimal number delimiter, dates. I am sure that some
languages would need special tests.
BTW: How is the localization used during test run?
I know that I select locale in the "Run Tests - Your Chosen Test Run"
If I select "de", will I see only the "de" test cases from the l10n
branch? I am not sure if it is important but it would be nice.
> > Case 1. Carry priority information with subgroups name
> This scheme would make it impossible to create a testrun for e.g. just
> P1 tests. Or just writer tests. I guess this is pretty much the same as
> what we have now, so all criticism from above paragraph applies here too.
Good point. We can't use this.
> > Case 2. Carry priority information with groups name
> This is probably OK if you don't mind having to create three testruns
> for each version
It is not ideal but acceptable.
> and be unable to create testruns for separate components
IMHO, this is fine.
> . I think you forgot locales here though. I assume that for
> each locale/priority there would be a separate testgroup?
I guess so. Hmm, that would create too many test groups. See my other
> > Case 3. Carry priority information with branches
> IMO, Case 3 is the worst case, because for each version of LibreOffice,
> you would have to create 4*3=12 testruns – one for each priority/test
> type combination.
I am not sure if it is the worst case because the solution 1 was bad as
well :-) Anyway, I think that 3 is not a way to go.
> My suggestion is to have a single branch, but carry Priority, Locale and
> Component information in the Testgroup, and to represent test type by a
> subgroup. That is, what is now a branch, would become a subgroup, and
> everything else would become a testgroup (see the attached PDF file).
Hmm, I am afraid that we would get too many testgroups. It produces 7
(General, Base, Calc, Draw, Impress, Math, Writer) testgroups for one
language. We have 109 localizations in sources => we could end up with
more than 700 test groups.
> This allows to:
> * create testruns based on priority, locale, component, or any
> combination of these
The question is what test runs we will want to create.
> * create a single catch-all testrun for a single version of LibO
would be nice
> * share the same General Functional Tests subgroup between testgroups
> designated for different locales (that is, you would create this
> subgroup once, and add it to all locale groups)
I am not sure what you mean with this. Could we share subgroups between
test groups in Litmus? Is it clearly visible or is hard to maintain and
BTW: What do you mean with the "Basis Functional Test"? We do not have
this subgroup in the current structure.
> * drop 75% of the testgroups (there would be about 28 of them initially)
> if/when proper testcase localization is implemented, still not rendering
> the remaining testgroups useless.
I am not sure if we really could drop them. Well, the question is if we
need to split between lang-dependent and lang-independent on the top
Heh, it seems that it is quite complex problem. I am going to write
another mail where I will try to summarize some ideas :-)
Thanks a lot for the valuable input.
More information about the Libreoffice-qa