[Libreoffice] concept for c++ based subsequenttests

[Markus Mohrhard]

Hey,

thanks a lot for your answers.

2011/11/30 Michael Stahl <mstahl at redhat.com>:
>> - having test files instead of code that fills the cells make it
>> easier to use the test data outside of the test ( e.g. debugging a
>> problem, having a testdocument with known result for some actions)
>
> hmm... am somewhat unsure about this: the ODF import uses the UNO API to
> a large extent, so it could be possible that the test breaks because the
> API used by the import is borked, before you get to the actual test of
> that same API that would point out directly the failing method (which is
> much easier to debug than a somehow wrong import).

At least in calc we have several tests for file import which are
in-build, so I think that using the import filter should be quite
safe. I'm sure it is not a perfect solution because we are no longer
only testing one aspect but on the other hand we have no clean tests
in any place. I hope that the c++ based tests won't randomly fail so
if they are executed at least regularly on the tinderboxes I would
favor a failure due to a filter problem over an unnoticed bug.

>
>> - only a little bit of code needed for a new test, most code is shared
>> with our in-build tests
>
> that is of course an advantage of your approach.
>
>> - porting old tests is straightforward
>> - does not need an installation
>> - needs much more makefile lines than the old tests ( there might be a
>> solution to this )
>
> seems makefile is so big that it didn't fit in the mail ;)

Yeah missed that. Should now be in master with the makefile.

>
> i am somewhat unsure if the current approach of having essentially a
> custom makefile for each test is the right way to go; sure you don't
> need a running office, but instead every test makefile needs careful
> maintenance so all the dependencies are mentioned (which are not always
> obvious), and custom rdbs and stuff.

I think for now one makefile per test is fine and as soon as we get
more and more c++ based tests maybe one makefile per module or group
related tests into one makefile. I think that is something that will
show up when there are more tests and there is need for a better
concept regarding makefiles. But I like the idea of one makefile per
module for c++ based subsequenttest.

>
> now if only we could run soffice from the solver directly...

A big advantage over running a full LibO installation is that you can
move these tests between the in-build unit test target and the
subsequenttests target. Ok, this point is only valid for sc, sw and sd
but that is already a huge advantage if you plan some rework in an
area you can easily add the relevant tests to the in-build tests and
don't need to run all tests.

>
>> The concept needs still a lot of love and is only intended as starting
>> point. I hope that this makes debugging subsequenttests easier and
>> writing new tests as easy as writing in-build tests.
>>
>> Do you have comments or suggestions?
>
> i second Stephan's request about not using qa/unoapi: that has a very
> specific meaning firmly tied to the qadevOOo mess.

Ok. I just used it because it looked correct for testing the uno api.
I'll create a new dir in sc/qa for the c++ based tests.

>
> hmmm.... i wonder if it would make sense to have UNO API tests written
> in Python: that should be much easier on the eyes than boilerplate-heavy
> C++/Java... and i think there is a need for tests at the UNO API level
> no matter how many core C++ level tests we have, because there is really
> no other way to find regressions in that area (nobody tests that
> manually), might as well try to maximize the productivity...
>

I planned to use this concept for uno tests. For most of the other
test areas I have already better test concepts. This really slow and
heavy concept is only needed if you plan to use uno api calls that
might use UI code or test the vba implementation. The advantage of a
C++ based test is that debugging is extremely easy, I'm not sure that
this is true for python based tests.

Regards,
Markus