[Libreoffice] subsequenttests now run headless

[Bjoern Michaelsen]

Hi Miklos,

lengthy answer to your innocent question ahead -- brace for impact.

On Tue, 29 Mar 2011 23:18:20 +0200
Miklos Vajna <vmiklos at frugalware.org> wrote:
 
> I'm sure this is a newbie question, but what is the relation between
> unit tests, smoketest and subsequenttests?

There is a whole lot of terms flying around wrt tests, which is rather
unfortunate.
There are:
  - unit tests (the real ones, somewhere in $MODULE/qa)
  - smoketest (in module smoketestoo_native)
  - subsequent tests including:
    - complex tests (mostly in $MODULE/qa/complex)
    - unoapi tests (in $MODULE/qa/unoapi and qadevOOo)
  - testtool tests

Unittests are run during the build. The smoketest tests the full
product so that it doesnt crash right upon start or while loading a
document. It requires a full installation for that of course.
Subsequent tests do various additional tests that require a full
installation. Complex tests are manually written "realworld scenarios"
that use a feature in the way intended on development. Unoapi tests
just do some very basic tests (setting a value, reading the value back,
checking if it is the same). They are no useful "realworld scenarios"
and highly synthetic. And finally there is the VclTesttool which
simulates mouseclicks and GUI events and hooks into the UI toolkit.

> If I'm not mistaken, the later two is integration test, while cppunit
> is about lowlevel unit testing - does that mean that we want to get
> rid of smoketest in the long run?

IMHO not at all. The list above is exactly in order of the
test usefulness (with real unittests being the most useful). However,
the real unittests are also the hardest to get: Some parts of code
require so much infrastructure (UNO, Config, GUI etc.) that you need to
do a lot of trickery to test it without a complete installation. That
is unfortunate because:

 - it is a lot of extra work.
 - the trickery might be fragile and hard to maintain.

The best solution would be to refactor your code to real small and
nifty units. But that is a lot of code change and for refactoring you
might want to have tests before you change (and break) everything. This
is were the complex tests come in IMHO:

You first write a "complex test" running real world scenarios on your
code area. It requires a full installation, but still it is better than
manually testing over and over again. Then you refactor mercilessly,
kill of useless dependencies and end up with a nifty small unit worth
the name "unit". Then you write a small unit test for the code that can
be run during the build, which should be easy (or at least easier) now.
You leave the complex test around because it doesnt hurt to have it (if
it fails, it might be worth a look).

The unoapi tests are still somewhat useful because of their code
coverage. But I dont think it is worth it to invest too much time into
writing new ones, when we can better write "real unittest" (and complex
tests -- to get real unittests).

As for the testtool tests -- I dont find them worth the effort. They
are very fragile (because they rely on timing and things like that) and
rarely are helpful in triaging bugs. There is one scenario I remember
them to be useful though: When migrating modules to the new build
system, they found missing resources sometimes. But that is a rather
special case and not really worth the long runtime and fragile results.

Running smoketest before pushing a nontrivial changeset is a good idea. 

tl;dr: I dont think we should get rid of subsequenttests or smoketest
as of now as they are cheap to keep. But whenever it is possible to
write "real unit tests" that should be preferred.

Best Regards,

Bjoern

P.S.:
BTW not all subsequent tests are in Java, some are in C++ too. Thus also
not every cppunit test is run during the build currently.

P.P.S.:
Here is how I personally usually run the tests:
unit tests:     on every build (doh)
smoketest:      before nontrivial pushes along with manual testing
                after a major rebuild because of changes on master
                  (some branch integration for example)
complex tests:  on the module I code along with manual testing
                twice a week (before going to lunch)
		on releases (starting with 3.4.X)
unoapi tests:   as complex tests
testtool tests: not at all unless forced by some policy   

-- 
https://launchpad.net/~bjoern-michaelsen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://lists.freedesktop.org/archives/libreoffice/attachments/20110330/1ddfc27b/attachment.pgp>