[Libreoffice-qa] test cases quality; was: Ubuntu/Canonical doing more manual testing for LibreOffice?

[Sophie Gautier]

Petr,

First, I don't take anything personal in your mail. I disagree with you 
but it's nothing personal :)

On 02/03/2012 17:26, Petr Mladek wrote:
> Sophie Gautier píše v Pá 02. 03. 2012 v 16:20 +0100:
>>> Another bunch of tests sounds like:
>>>
>>> 	+ Translation check of creating a new database
>>> 	+ Translation check when creating a table in a database
>>> 	+ Translation check for Formula Editor
>>
>> Well, I don't think you get the purpose of what Litmus was done for. It
>> was for community testing at large, so very easy and short tests to
>> bring interest to the testing. It should have help also localizer to
>> test there version. Just as we did by the past and it worked well. Some
>> spend only 30mn others more that 3 hours because the online tests was
>> only the very basis of larger tests with a set of documents. So it's
>> more about the life of a team, than only a basic test. Unfortunately we
>> don't have the good tool here and no money to develop what could suite
>> our needs. Mozilla was developing a tool but it's not yet done either.
>
> I appreciate that you want to teach people using Litmus. Though, I am
> afraid that you did not get my point.

I don't want to teach them using Litmus, I want them to get an interest, 
get fun and don't feel harassed by the task.

>
> Please, read the above mentioned test cases. One test describe how get
> into one dialog and asks to check that all strings are translated.
> Another test cases describes how to reach another dialog where the
> strings need to be checked
The check for the translation is a second purpose of the test, the first 
purpose is to check the basics functionalities such as Save as, Open, 
Copy, Past... etc.
>
> IMHO, there are hunderts or thousands of dialogs. IMHO, we do not want
> a test case for every single dialog. We do not have enough people who
> could create, translate, and process all such test cases.

We are testing functionalities and by the same way are checking for 
basic i18n conversion (numbers, accentuated characters, date, size of 
the string...)
>
> Also I am not sure if would be effective to use Litmus for this type of
> testing. It might take few seconds to check that all strings are in a
> given language. It might take longer time until you enter your result in
> Litmus and select another test case.

Litmus should be an entry for approaching QA for the community at large 
i.e. no language barrier, no technical barrier, a team behind to guide 
you further in more complex testing. Unfortunately, it's not a tool 
adapted to our needs.
>
> IMHO, we could do much better job here. If we have strings translated in
> pootle and the build works correctly, all translated strings are used.
> By other words, if you have translation for 1000 dialogs in pootle, it
> is enough to QA only 1 dialog. The strings are extracted from pootle by
> a script and applied in sources by another tool. If one string is used,
> others are used as well[*].

As said, I'm not speaking about translation. The contents of the test 
may confuse you when it speaks about localization, but it's only a 
second purpose of the test, a "*while you are here*, please check that 
the dialog has the good special characters in your language"
>
> You might say that you need to check layout of the strings that they are
> not shrinked. Well, we need not check all strings here. It might be
> enough to check only strings that look risky (translation is much
> longer) than the original string.

No, it's not enough, because most of the time, the team doing the 
translation is one person only, so you can't remember where and when the 
translation is longer than the original, and for some languages it's 
always true.
>
> You might say that we should check quality of the translation. I mean if
> the translation makes sense in the context of the given dialog. Well,
> this is not mentioned in the current test case. Also, I am not sure if
> it is worth the effort. We do not change all strings in every release.
> So, we do not need to check all translations.

When you see the amount of strings for the number of people doing 
translation, having a proof reading of the dialog during QA is not a 
luxury ;) But I agree, as said it's not the first aim of the tests
>
>
>>> Of course, we need to check that the application is translated but we
>>> can't check every dialog manually.
>>
>> We had that by the past with the VCLTestool.
>
> Hmm, how VCLTesttool helped here? Did it checked that a string was
> localized? Did it checked if a translation was shrinked or confusing?

It took a snapshot of each dialog, menu, submenu, etc. When you want to 
reach a certain amount of quality for you version, it was very useful 
because you were sure that everything was checked. I don't say that you 
run it on each version but I did it on each major OOo versions.
>
>>    Instead of the above particular
>>> dialogs, we should check that different elements are localized, for
>>> example:
>>>
>>> 	+ "File/New" menu - because it consists of optional components
>>>                             that are added from xml registry files
>>>    	+ main menu and one submenu
>>> 	+ a dialog with tabs, check boxes, combo boxes, itemized list,
>>>             and other elements
>>> 	+ help - because it using another technology than the other
>>>                    dialogs
>>> 	+ KDE/GNOME safe dialog because they are done another technology
>>>                    as well
>>> 	+ extensions - because the translation is done slightly
>>>                       different way
>
> [*] There are several type of strings which are processed different way.
> The above list mentions the basic categories. I suggest to test examples
> of the categories instead of every single dialog.
>
>
>>> If one submenu is localized, the other submenus should be localized as
>>> well if the strings are in pootle.
>>
>> It's not about localization only (but it's good for CTL and CJK) , but
>> also about the design of the dialog that allow to see the whole string
>> and then adapt the dialog or the l10n. It's not about to see if it
>> works, it's about the quality of the l10n and the design.
>
> I agree that we need to test this but we are back in the current test
> cases. They do not mention CTL/CJK problems. They do not ask for design
> check. They do not concentrate of functionality that is really affected
> by CTL/CJK.

Because each team has to adapt the test in his language, the basis in 
English doesn't mention every specificities.
>
> I am neither localize-person nor professional QA guy. I have just
> feeling that we could do the testing more effectively and the test cases
> should teach people how to do it. IMHO, this is not the current state.

Yes, you're right. But keep in mind that to teach people in their spare 
time, they need to enjoy it. It needs to be a step by step learning, 
growing interest as well as knowledge at the same time. And don't forget 
the fun too. There should be very simple test cases and more complex 
ones. Simple samples of document and much more complex ones. Defined 
period of test to create a dynamic in the group, with visible results 
and visible recognition, etc...
>
> IMHO, it would be easier that start with functional tests rather than
> entering hunderts of the "same" translations tests.

yes, of course :) but I think you get what I was meaning. You may see 
localization of a test as repeating the test, when it's offering 
somebody to come in with no other burden that the joy of participating 
in his language and with his basic skills. Accept to waste some time 
with less effective test but allow to more people to participate is what 
is behind that tool.
>
>
>>> IMHO, we need to discuss what test cases make sense and create a
>>> reasonable test cases first.
>>>
>>> We are still looking for an experienced QA guy who could step in, teach
>>> people and drive this forward.
>>
>> So lets wait for that guy.
>
> Sophie, I feel something negative in this sentence. Please, do not take
> this mail personally. I know that you do much much work for this
> project.

this is not negative, it's a bit fatalist because your technical vision 
is right and effective in a matter of testing, but it doesn't give 
access to a wide range of people with very low skills but full of energy 
that will request us a little (or more) time but for a strong feedback. 
And I don't have the energy or the time to demonstrate that :)
>
> I try to show QA people some interesting directions when I found time.
> Unfortunately, I have many other tasks in the release process and not
> enough energy left for this interesting QA area.

As we all are. But anyway the work has to be done :-)

Kind regards
Sophie
-- 
Founding member of The Document Foundation