[Libreoffice-qa] Test case naming

[Rimas Kudelis]

Hello Petr, and everyone,

2011.11.11 22:14, Petr Mladek rašė:
> The list of existing test cases looks the following way in Litmus:
>
> id   # testcase summary                                      # subgroup
> -----------------------------------------------------------------------
> 1066 # g001 - Installing LibreOffice                         # General
> 1061 # g002 - Uninstalling LibreOffice                       # General
> 1053 # g003 - First launch of LibreOffice                    # General
> 1063 # g004 - Creating a new document                        # General
> 1048 # w002 - Importing MS Word documents                    # Writer
> 1052 # w003 - Exporting ODF text document to MS Word formats # Writer 
> 1049 # w004 - PDF export of text docs                        # Writer
> 1054 # w001 - Creating a new text document                   # Writer
>
> See also the screenshot at http://download.go-oo.org/qa/test-case-list.png
>
>
> I see two problems here:
>
> 1. The test cases are ordered by the id, so the first writer test case
>    is the last in the list.

No, they are not ordered by id. Actually, I've just digged, and the
order is as specified for subgroups inside a testgroup, and then as
specified for testcases inside each subgroup.

If you look at what you pasted (and at your image) again, you'll notice
that #w001 is the only testcase that wasn't shown in its expected
position, and that is because it wasn't positioned correctly inside the
subgroup. I've just corrected its position and it's now shown where
expected.

>    I wonder how complicated would be to hack Litmus to sort it by
>    the testcase summary.

This is thus irrelevant, cause there's nothing to modify. :)

> 2. I am not sure what is the meaning of the numbers 001, 002, 003.
>
>    It looks like they define the order in which we should process the
>    test cases. If this is true, it does not look ideal:
>
> 	+ if we do another important test case, we will need to rename
>           all less important test cases to keep the right order
>
>         + test cases will be checked by many people; we can't force them
>           to do it in an exact order; the result would be that all
>           people will test the same test case in parallel
>
>           Hmm, we need to encourage people to do the test cases in
>           random order. We still should somehow prioritize the test
>           cases.

I personally don't like current naming with ugly prefices at all, but
it's Yifan's call, and I suppose he has a good reason for that naming
scheme. However, I'm afraid we can't randomize testcase order by
default. Currently, this would probably have to be done manually each
time a relevant subgroup is updated. That would be a PITA. On the other
hand, it's not really that bad. You can still run the tests in random
order, but you will always see them in the specified order.

>     I suggest to split test cases into several levels by priorities:
>
>     	P1 - highest: used for very basic tests, e.g. app can be
>              installed; it starts; is able to load/save some test
>              documents; so it a kind of smoketest
>
>         P2 - high: test very common functionality that is used by most
>              users. e.g. able to write text, insert picture; draw
>              elements; create table; use function in calc; create graph,
>              run presentation
>
>         P3 - medium: test common functionality that is used by typical
>              a bit experienced office user, e.g. create borders around
>              tables; do animation between slides; modify text style;
>              modify master slide page;
>
>         P4 - low: test functionality for hi-tech users, e.g. writing
>              macros, using Calc solver, complex operations with data
>              bases
>
>     I suggest to use the names:
>
>     p1g - <summary of a P1 global test>
>     p1w - <summary of a P1 Writer test>
>     p2g - <summary of a P2 global test>
>     p2w - <summary of a P2 Writer test>
>
>     Then we will have all p1 test cases listed before p2 test cases.
>
>
> What do you think?

Prioritizing is probably a good idea, but like I said, random order
would require some Litmus modification. While certainly possible (and
probably quite easy), I'm not sure that is what we indeed want.

Rimas