[Libreoffice-qa] db test VM

[Stephan Bergmann]

On 10/31/2011 10:19 AM, Petr Mladek wrote:
> What is your opinion, please?

"Hard to tell without trying it out," I'd say.  Maybe pick some random 
test now done via testtool and see if it can be ported to 
subsequenttest's "pure UNO" approach and is better afterwards.

> My feeling is that many people do not like testtool because it is:
>
> 	+ gives random failures:
> 		+ sleeps and waits everywhere in the test code

The root problem here probably is that in some cases appropriate 
notification channels are missing, so test code inserts little waits. 
Some other places are probably "test didn't work first, inserted a 
sleep, test then worked, no idea why" that should rather be analyzed fully.

> 	+ hard to debug:
> 		+ unclear error messaged
> 		+ complex mapping between the elements in the testtool
>                    code and LO code (icons, dialogs)
> 		+ need to run the test several times to understand what
>                    happens there =>  quite time consuming
> 	+ hard to maintain:
> 		+ menu entries are accessed by the position in menu =>
>                    new item breaks tests

On the other hand, this did find errors in the past.  ;)  (Some menus or 
drop-down lists maintained items in the order as provided by configmgr, 
relying on configmgr providing them in a specific order; after the 
configmgr rewrite, the items were in a different order, and the actions 
associated with the items no longer matched the items' text...)

> IMHO, the best solution would be to involve developers into maintaining
> the tests. We need some technology that they would like and accept.

Yep.  Ideally, such higher level tests should be understandable, 
writeable, and maintainable by both QA and Dev folks.  testtool is 
probably something that neither camp particularly likes.

Stephan