[Libreoffice-qa] GUI automation

[Michael Meeks]

Hi Vladimir,

	First - great to see you here :-)

On Wed, 2013-03-13 at 16:17 +0100, Vladimir Benes wrote:
> we were discussing recently how to improve testing of LO's GUI. We found 
> out that situation if more than bad.

	:-)

> There are several options how to approach this. The first is LDTP [1] 
> and the other one (in my opinion far better) is Dogtail [2]. If LO uses 
> standard gtk widgets so they're a11y friendly we can easily access them. 
> This works for accessing menu items, dialogs and so on.

	We used to have a huge chunk of code that did this GUI testing. The
general consensus is/was that it is unreliable, and broadly worthless -
breaking on minor UI changes, requiring lots of maintenance, incredibly
slow to run, etc. We removed it all :-)

> For comparing visual things like pictures, font alignment, etc there is 
> Sikuli [3] framework actively developed at MIT that can be used for this.

	IMHO it would be far more useful to develop unit tests that are run
during the build; and/or to write load/save/load tests such that we
check that our filters are not loosing data left/right as they load and
save :-) We have a lot of such tests already, we also have some layout
test code that dumps the layout tree in writer to try to check that the
output is the same, and more pieces in that vein.

> Is there anything where we can start or at least help with? What I would 
> expect is at least some priority one functionality that should be 
> covered first or so.

	Caolan prolly has some more good ideas around what is needed here; and
we could use more man-power and build-hardware of course. Markus is an
expert in the unit testing area (and elsewhere) and has a rather nice
tool to run a ton of bug-documents through the code: it'd be great to
get that hosted, running regularly, and the bugs filed / fixed alongside
the tinderboxes.

	Then of course, there is running that same (15k document) test suite
regularly under valgrind - that would also be useful ;-)

	So - IIWY I'd do GUI testing as the very last thing in the stack of
other more useful things that can be invested in to improve quality.

	Does that help ?

		Michael.

-- 
michael.meeks at suse.com  <><, Pseudo Engineer, itinerant idiot