[Libreoffice] Contributing test cases

[Caolán McNamara]

On Thu, 2010-11-18 at 12:16 +0100, Gioele Barabucci wrote:
> Hello,
> 
> in order to report a subtle bug [1], I had to create a minimal failing 
> test case document.
> 
> Is there a way to contribute this kind of testcases, and assertions on 
> which results one should expect, to a testsuite so that future changes 
> to the LibreOffice code can be tested against these well-known bugs?

We really should try and organize this alright. I know there are some
adhoc collections of documents here and there, e.g.scratch/sc-vba, but
we need to get something organized and better automated IMO.

What I'd like to see, and I'm still working on a few pieces, is to get
basic cppunit build-time tests for the major apps together, e.g. a
basic calc one is working under Linux and I've added another super-basic
one I haven't enabled yet for starmath. As part of those tests I think
I'd like to have a least one load/save for each file format, maybe,
depends on the practicability of that. First though I need to get the
smoketest reliable on windows at least.

> In my case I'd need to describe something like «type this, type that, 
> apply this style; save; in the resulting XML file there should not be a 
> space between this element and this other element».

Something like that might be suitable for a build-time cppunit test. But
either way we definitely should have slower automated tests that run
separately. I feel the "testtool" thing is a bit of a disaster really,
needs too much continual poking and adding timeouts etc to keep it
going. There other random grabbags of stuff in qadevOOO, then there's
the subsequent tests etc. What I think I'd like to see is

1) basic cppunit tests take place during the build, keep us honest and
basic functionality working at all times. Have a few of these at the
moment.
2) smoketest run at the end of the build, at least by the buildbots to
make sure that works. This needs a little bit more love, at the moment
I've some enthusiasm that I may have found some of problems that makes
it die horribly at exit time for some people.
3) a bit optional module/repo filled with masses of test cases with a
simple "make" target that gets run at release time and/or whenever qa
folk want to test the build.

C.