[Libreoffice-qa] db test VM (was: automated testing of Base)

[drew]

On Mon, 2011-10-10 at 12:36 +0200, Petr Mladek wrote:
> Hi,
> 
> I add Yi Fan into CC who actively works on LO QA.
> 
> Michael Meeks píše v Pá 07. 10. 2011 v 20:49 +0100:
> > Hi Drew,
> > 
> > On Fri, 2011-10-07 at 13:28 -0400, drew wrote:
> > > I read over some material regarding the opensuse.openQA goings on ...
> > 
> > 	Wow, encouraging research there.
> > 
> > > http://susestudio.org/a/EvUJ20/lo-sbs-libre-34x
> 
> Looks pretty interesting. I have few questions: 
> 
> Is is supposed for testing of the openSUSE LO packages or the plain LO
> build?

*chuckling*...good question - it is currently carrying the Suse(ified)
version of LibO. Vanilla (plain) LibO would make more sense and my
preference. 

> Is the testing automatic or manual?

It doesn't do any testing *yet* (I doubt [know, as the config script
isn't written yet] it will even load properly today) - but when I first
thought about something like this the plan was to have a set of
automatic tests and then have it as a resource for manual testing. 

> 
> I wonder how to synchronize it with the other testing effort. Note that
> test cases and test runs for the plain build are organized via Litmus,
> see http://wiki.documentfoundation.org/QA#Function_test .

Yes, I'm aware of that tool - read over the wiki pages, again, this
weekend. It seemed to me that there would be a connection between the
two, or to say that I would use the litmus system to document the test
cases and the VM to execute them. 

Actually in the first incantation of thought I had planned on having a
very pronounced split between a set of back end images and front end
images so that one could build a testing system to cover multiple vendor
releases of OO.o [1] 

- I did get to where I had a number of scripts working for some basic
test, the scripts where all OOoBasic and exercised a set of OO.o API
functions (though OpenSolaris proved bothersome and then I kind of lost
my cool on the whole OO.o experience, but that is neither here nor
there)

- I still think this is a reasonable way to go, though I might change
the scripting language from Basic to something else perhaps, in the end
the API is the constant.

- This API testing is IMO certainly useful but for GUI stuff I did not
have a good solution. Sounds like the OpenQA system could step in here.
I have some old experience with automated GUI testing tools and I have
to say they never left a good taste in my mouth, primarily because of
false positives, but that experience is old and hopefully much of that
problem has been fixed by folks much smarter then me.


>   Test cases
> for the openSUSE LO packages are in Testopia that is part of
> https://bugzilla.novell.com. I am not sure if people outside SUSE have
> access to it, though.
> 
> Also I wonder where to report bugs. I would prefer
> https://bugs.freedesktop.org/ if the bugs are in the plain build as
> well.

Where else? (so yes)

> 
> > 	I wonder how we get the latest LibreOffice versions into there ?
> > whether we build them, or install packages or something ? Bernhard
> > and/or Petr - what is the easiest way to get say a nightly snapshot into
> > an openSUSE virtual machine for automated / openQA testing ?
> 
> One possibility would be to use night builds from the plain build.
> Alternatively, I could try to produce nightly builds in BS. I already
> have many useful scripts at hands.
> 

so bottom line - I'm willing to put some time and effort into this - and
open to suggestions on how best to integrate into what others are
doing..

Best wishes,

//drew

[1] http://drewjensen.typepad.com/blog/2009/12/meet-sybil.html