Fwd: Code Coverage LibreOffice

Tue Nov 3 12:11:12 UTC 2020

Dear LO developers,

A simple/naive idea:
Loading ODT files into LO while doing a coverage test.
The source code coverage DIFF will reveal the source code being used
exclusively for the ODT feature DIFF of the two documents.

For example:

   1. Load a very simple test document, like a single paragraph with "Hello
   World" with LibreOffice and during load doing a code coverage test to
   identify all the source code, which is being used for loading the ODF test
   document.
   2. Enhance the simple test document by a single feature, like making the
   text bold and do the loading & code coverage again.
   3. The difference (DIFF) between the two source code coverages
   represents source code exclusively relevant to the feature bold.
   4. Future commits changing source code of such an "ODF feature area"
   would trigger the load/save roundtrip test of these special ODF test
   document(s).
   *The advantage is to have a relevant minimum regression test not taking
   long to execute.*

I already interviewed Maarten, yesterday, who wrote some years ago scripts
to run the coverage tests on LO, but they seem not to be executed for a
long time:
https://dev-builds.libreoffice.org/lcov_reports/master-2015-06-10_23.49.39/

Is there some LO build VM available that can be extended to reactivate the
code coverage tests, so interested parties might follow this approach?
Especially, I am longing for a "black box script" where I can give a text
file with a list of ODF files as an input parameter to be code-cover-tested
and receive for each input ODF the code coverage as a text file. Is there
any script that can e used?
In post-processing, the source code DIFF would than be created afterwards.
Shouldn't be too difficult (famous last words)... ;-)

Is anyone interested to join this adventure? :-)

Cheers,
Svante

---------- Forwarded message ---------
Von: Maarten Hoes <hoes.maarten at gmail.com>
Date: Mo., 2. Nov. 2020 um 22:35 Uhr
Subject: Re: Code Coverage LibreOffice
To: Svante Schubert <svante.schubert at gmail.com>
Cc: Michael Stahl <mst at libreoffice.org>

Hi Svante/Michael,

I'm glad to be of help, if I can be. :)

First of all, a disclaimer: I'm not a developer. I can write Unix/Linux
shell scripts, and used to be a professional Unix/Linux system
administrator, but I couldn't write a 'hello world' program in C or C++ if
my life depended on it.

Perhaps I should start with some basics. When talking about 'code
coverage', people generally mean 'measure what source code gets executed'
(generally by the execution of a test suite), so that you can see how much
and what parts of the source code gets covered by the test suite.

There is a wiki page [1] that describes how to set up lcov/gcov on Linux
for the LibreOffice codebase, but I can't tell how accurate or up to date
that wiki page currently is. Years ago, I wrote a shell script to automate
that [2] process. If I recall correctly, it used to be run automatically
(daily ?) by Jenkins, but the host/vm the job ran on became unavailable
ages ago, and no-one seemed to notice or care enough to restore it. Old
examples of what the generated reports look like can be found on the
dev-builds.libreoffice.org website [3].

The process basically looks somewhat like this:

- Compile your code using additional flags, so that a record will be kept
of what code gets executed.
- Run the test suite (or whatever else of which you want to see what source
code gets executed by running it).
- Generate a human readable report that shows what sourcecode got executed.

Now, on to your specific questions:

1. Yes, lcov/gcov code coverage will show you all the source code lines
that get executed by loading your 'Hello World' test document.
2. Yes, loading the same document multiple times will execute the exact
same source code every time you do it, provided you don't change any of the
conditions (like changing the source code in between runs).
3. Yes, I believe that if you take your 'Hello World' test document, and
make that text bold, then the difference between the two runs should be the
additional code that gets executed by making the text bold.

One thing to note though, as it may not be immediately obvious: When you
start LibreOffice, and open a document, a lot more code will get executed
than 'just' the code needed to display the text in the document. All of
this additional code will get marked as having been executed in the report.
I know of no way around that. So I am assUming (but you know what happens
when we assUme) that you need to be intimately familiar with the entire
LibreOffice codebase in order to differentiate between 'this is code that
gets executed on startup, this is code needed for the GUI, this is code
needed for parsing the document' and 'this is the code for the bold feature
I want to test'. So I'm not so sure if this gets you specifically what you
are looking for. (PS: There is a way to filter out / omit source code that
you are not interested in in the final report, but this filtering is done
on a per directory-basis. So you can for example filter out
'src/but/not/this/directory', but then you need to be very sure up-front
that the code you are looking for is not contained in that directory. This
also won't help you if you have 'code you are interested in' and 'code not
interested in' that are both located in the same directory).

I hope this may assist to get you started. As I may have raised more
questions than I attempted to answer, please feel free to let me know, and
I will try my best to be of further assistance.

- Maarten

[1]
https://wiki.documentfoundation.org/Development/Lcov

[2]
https://git.libreoffice.org/buildbot/+/refs/heads/master/lcov-report/

[3]
https://dev-builds.libreoffice.org/lcov_reports/

On Mon, Nov 2, 2020 at 8:55 PM Svante Schubert <svante.schubert at gmail.com>
wrote:

> Hi Maarten,
>
> Michael Stahl - who is as me a TDF member - dropped your name after the
> OASIS ODF TC call, in which we are both participating.
> I was curious about LO Code Coverage, but I am more working on ODF than on
> LibreOffice itself, for instance, the https://github.com/tdf/odftoolkit
> There is a little experiment I have in mind, my hope is that you might get
> curious enough to join it ;-)
>
> The simple idea:
>
>    1. Load a very simple test document, like a single paragraph with
>    "Hello World" with LibreOffice and during load doing a code coverage of all
>    the source code being used for loading the test document.
>    2. Load the simple test document multiple times, is it always the same
>    code coverage? (As I said I am new to LO Code and do not know the result)
>    3. If it is the same (or changes within a predictable range), enhance
>    the simple test document by a single feature, like making the text bold and
>    do the loading & code coverage again. Is the difference between the two
>    coverages the feature bold?
>
> The question is: Is it possible to create an ODF feature <=> source code
> relation by doing the above?
> The advantage, if certain parts of source code are being changed, we have
> better insights on the ODF Testdocuments we are in need.
>
> The reason why I am asking is that the ODF Toolkit has a library ODFDOM,
> which is able to transform an ODT document
> <https://github.com/tdf/odftoolkit/blob/master/docs/docs/presentations/character-styles.odt>
> into an equivalent list of user changes (in JSON)
> <https://github.com/tdf/odftoolkit/blob/master/docs/docs/presentations/character-styles.json>
> as if the document was created from top to bottom. By this, we are able to
> define more precisely the feature delta of the coverage scenario.
>
> In the upcoming year, I am trying to do several things:
>
>    1. Add to the OASIS ODF specification these user changes (you may call
>    them as well "ODF user features" or "ODF user semantics" or simply "ODF
>    feature")
>    In addition, to add a semantic dictionary.
>    2. Refactor the work on the ODFDOM of ODF Toolkit...(many details)...
>
> I would be very curious to set-up an environment, which I can use as a
> black box, providing a document as input and receive the code coverage as
> output!🤗
> Would you be so kind to assist me in that, Maarten? As I have frankly
> little idea where to start ;-)
>
> Best regards,
> Svante
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice/attachments/20201103/aec07818/attachment.htm>