[Libreoffice-qa] Minutes of ESC call: 2015-07-30

[Markus Mohrhard]

Hey,

On Sun, Aug 2, 2015 at 9:01 PM, Norbert Thiebaud <nthiebaud at gmail.com>
wrote:

> On Sun, Aug 2, 2015 at 11:25 AM, Markus Mohrhard
> <markus.mohrhard at googlemail.com> wrote:
>
>
> >> AI:     + will have a look at the CppUnit to implement 'expected
> failure'
> >> (Jan-Marek)
> >>             + Cpp logs are e.g. in
> >> workdir/CppunitTest/sal_rtl_math.test.log
> >
> >
> >
> > That is already possible with cppunit. Instead of using CPPUNIT_TEST use
> > CPPUNIT_TEST_FAIL which tells cppunit that the test is expected to fail
> with
> > a cppunit exception being thrown (it is extensively used in the cppunit
> > internal unit tests). The test will start to fail when none of the
> asserts
> > fail anymore. Keep in mind that it might be dangerous to use this with
> more
> > than one assertion as an unexpected one might fail.
> >
>
> That is not exactly what tha aim is...
> the aim is to have a failure be repport as such and not stop
> everything.. when a test is tagged as _can fail_ or something
> this is when a test is added before the fix
>

So you want to have this as non-fatal tests. Ok, that is currently missing
from cppunit but at least with the CPPUNIT_TEST_FAIL you can add tests
before the bugfix and just change it from CPPUNIT_TEST_FAIL to CPPUNIT_TEST
when the fix has been implemented. So it is a compromise as it is already
there and we can have a look how difficult it would be to properly
implement non-fatal tests in cppunit.


>
> also the minutes claim that cppunit log exist.. it is true they do,
> but they are not exploitable.
> I would like to be able to repport at the end of the build a nice recap
>
> <module> <section> <nb_of_test> <#sucess> <#skipped> <#failed>
> <for each failed
> <space><testname> : FAILED (<optionally one line reason>)
>
> the minute point to
> workdir/CppunitTest/sal_rtl_math.test.log
> which contains:
> OK (3)
>
> but  writerperfect_writer.test.log fro instance
> contain 12027 lines.. most of it random trace messages, and I could
> not find any way to parse (at all, even less reliably) that thing to
> extract any useful information.
>
> maybe it is just a matter of using
>
> http://cppunit.sourceforge.net/doc/1.8.0/class_cpp_unit_1_1_xml_outputter.html
> and then adding a (optional) post processing step in the build to
> generate a nice summary....
>


It is even easier. You can add another TestListener in
sal/cppunittester/cppunittester.cxx that logs all failures, and executed
tests. Of course as there is no support for non-fatal tests right now you
can't log them. It would at least give you an information how many tests
are executed. I think there is no post-processing necessary as you get all
the information the the xml outputter has already in the listener and
process is directly there.

Regards,
Markus

>
> Norbert
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/libreoffice-qa/attachments/20150802/a3f57e8f/attachment.html>