[Libreoffice-qa] test cases quality; was: Ubuntu/Canonical doing more manual testing for LibreOffice?

[Petr Mladek]

Sophie Gautier píše v Pá 02. 03. 2012 v 16:20 +0100:
> > Another bunch of tests sounds like:
> >
> > 	+ Translation check of creating a new database
> > 	+ Translation check when creating a table in a database
> > 	+ Translation check for Formula Editor
> 
> Well, I don't think you get the purpose of what Litmus was done for. It 
> was for community testing at large, so very easy and short tests to 
> bring interest to the testing. It should have help also localizer to 
> test there version. Just as we did by the past and it worked well. Some 
> spend only 30mn others more that 3 hours because the online tests was 
> only the very basis of larger tests with a set of documents. So it's 
> more about the life of a team, than only a basic test. Unfortunately we 
> don't have the good tool here and no money to develop what could suite 
> our needs. Mozilla was developing a tool but it's not yet done either.

I appreciate that you want to teach people using Litmus. Though, I am
afraid that you did not get my point.

Please, read the above mentioned test cases. One test describe how get
into one dialog and asks to check that all strings are translated.
Another test cases describes how to reach another dialog where the
strings need to be checked.

IMHO, there are hunderts or thousands of dialogs. IMHO, we do not want
a test case for every single dialog. We do not have enough people who
could create, translate, and process all such test cases.

Also I am not sure if would be effective to use Litmus for this type of
testing. It might take few seconds to check that all strings are in a
given language. It might take longer time until you enter your result in
Litmus and select another test case.

IMHO, we could do much better job here. If we have strings translated in
pootle and the build works correctly, all translated strings are used.
By other words, if you have translation for 1000 dialogs in pootle, it
is enough to QA only 1 dialog. The strings are extracted from pootle by
a script and applied in sources by another tool. If one string is used,
others are used as well[*].

You might say that you need to check layout of the strings that they are
not shrinked. Well, we need not check all strings here. It might be
enough to check only strings that look risky (translation is much
longer) than the original string.

You might say that we should check quality of the translation. I mean if
the translation makes sense in the context of the given dialog. Well,
this is not mentioned in the current test case. Also, I am not sure if
it is worth the effort. We do not change all strings in every release.
So, we do not need to check all translations.


> > Of course, we need to check that the application is translated but we
> > can't check every dialog manually.
> 
> We had that by the past with the VCLTestool.

Hmm, how VCLTesttool helped here? Did it checked that a string was
localized? Did it checked if a translation was shrinked or confusing?

>   Instead of the above particular
> > dialogs, we should check that different elements are localized, for
> > example:
> >
> > 	+ "File/New" menu - because it consists of optional components
> >                            that are added from xml registry files
> >   	+ main menu and one submenu
> > 	+ a dialog with tabs, check boxes, combo boxes, itemized list,
> >            and other elements
> > 	+ help - because it using another technology than the other
> >                   dialogs
> > 	+ KDE/GNOME safe dialog because they are done another technology
> >                   as well
> > 	+ extensions - because the translation is done slightly
> >                      different way

[*] There are several type of strings which are processed different way.
The above list mentions the basic categories. I suggest to test examples
of the categories instead of every single dialog.


> > If one submenu is localized, the other submenus should be localized as
> > well if the strings are in pootle.
> 
> It's not about localization only (but it's good for CTL and CJK) , but 
> also about the design of the dialog that allow to see the whole string 
> and then adapt the dialog or the l10n. It's not about to see if it 
> works, it's about the quality of the l10n and the design.

I agree that we need to test this but we are back in the current test
cases. They do not mention CTL/CJK problems. They do not ask for design
check. They do not concentrate of functionality that is really affected
by CTL/CJK. 

I am neither localize-person nor professional QA guy. I have just
feeling that we could do the testing more effectively and the test cases
should teach people how to do it. IMHO, this is not the current state.

IMHO, it would be easier that start with functional tests rather than
entering hunderts of the "same" translations tests.


> > IMHO, we need to discuss what test cases make sense and create a
> > reasonable test cases first.
> >
> > We are still looking for an experienced QA guy who could step in, teach
> > people and drive this forward.
> 
> So lets wait for that guy.

Sophie, I feel something negative in this sentence. Please, do not take
this mail personally. I know that you do much much work for this
project.

I try to show QA people some interesting directions when I found time.
Unfortunately, I have many other tasks in the release process and not
enough energy left for this interesting QA area.


Best Regards,
Petr