[Libreoffice-qa] GUI automation

Markus Mohrhard markus.mohrhard at googlemail.com
Wed Mar 13 18:05:37 PDT 2013

Hey Vladimir,

2013/3/13 Michael Meeks <michael.meeks at suse.com>:
> Hi Vladimir,
>         First - great to see you here :-)
> On Wed, 2013-03-13 at 16:17 +0100, Vladimir Benes wrote:
>> we were discussing recently how to improve testing of LO's GUI. We found
>> out that situation if more than bad.

So I agree here with you but sadly I currently have no good idea how
to improve that. One idea I had a bit more than one year ago was to
use the uno commands for menu entries and an annotaion in the src
files for ui elements but I discarded it for several reasons. However
with the ui files it might finally be easily possible to follow this
idea with the ui files.

But please explore all of the concepts that you can think of. Our
codebase is so complex that some may work and others will reveal
either problems in our code or in the test concept. I'm happy to help
with any problems you have in implementing new test concepts.

>> There are several options how to approach this. The first is LDTP [1]
>> and the other one (in my opinion far better) is Dogtail [2]. If LO uses
>> standard gtk widgets so they're a11y friendly we can easily access them.
>> This works for accessing menu items, dialogs and so on.
>         We used to have a huge chunk of code that did this GUI testing. The
> general consensus is/was that it is unreliable, and broadly worthless -
> breaking on minor UI changes, requiring lots of maintenance, incredibly
> slow to run, etc. We removed it all :-)

I agree with Michael that the old GUI testing code was horrible and
unreliable and my big fear is that our horrible accessibility code
will not make it easier to use existing GUI testing frameworks but I'm
happy if someone prooves me wrong. I have no idea how IA2 and the new
widget dialogs play into that. Michael and Caolan may know better
which consequences they have on our accesibility code.

>> For comparing visual things like pictures, font alignment, etc there is
>> Sikuli [3] framework actively developed at MIT that can be used for this.
>         IMHO it would be far more useful to develop unit tests that are run
> during the build; and/or to write load/save/load tests such that we
> check that our filters are not loosing data left/right as they load and
> save :-) We have a lot of such tests already, we also have some layout
> test code that dumps the layout tree in writer to try to check that the
> output is the same, and more pieces in that vein.
>> Is there anything where we can start or at least help with? What I would
>> expect is at least some priority one functionality that should be
>> covered first or so.

So a few things on my list of untested functionality that is not
related to GUI testing and has higher priority on my list:

* Copy & paste
* Draw is totally untested
* Improved export tests: optional validation of the exported files (ODF/OOXML)
* Porting the failing API tests to C++ and re-enable them
* Exploring and integrating test documents from different providers:
gnumeric, abiword, calligra, OpenOffice
* Performance tests

>         Caolan prolly has some more good ideas around what is needed here; and
> we could use more man-power and build-hardware of course. Markus is an
> expert in the unit testing area (and elsewhere) and has a rather nice
> tool to run a ton of bug-documents through the code: it'd be great to
> get that hosted, running regularly, and the bugs filed / fixed alongside
> the tinderboxes.

That is mostly blocked on my side. There are a few nasty memory bugs
also in my local build that either show another memory corruption in
our code/toolchain or an error in my concept. I hope to find some free
time soon to look into these reports.

>         Then of course, there is running that same (15k document) test suite
> regularly under valgrind - that would also be useful ;-)
>         So - IIWY I'd do GUI testing as the very last thing in the stack of
> other more useful things that can be invested in to improve quality.

I'm a bit more enthusiastic about the idea of GUI testing but there
must be someone who explores the different ideas and implements a
prototype. It took us around half a year until the propotype for
import tests was copied by other developers and found widespread use
in Libreoffice. So in my opinion the first step is to explore all of
the ideas and implement a reliably working prototype that can be used
to promote the idea.

I know that it takes time to implement and promote a new testing idea
and I hope that GUI testing will help to have a testing concept for
more of our fixed bugs.

So if you want to spend a bit of your time on exploring the different
concepts you mentioned you have my support and I'll try to help
wherever I can.


More information about the Libreoffice-qa mailing list