enhancing perfcheck - Proof of concept & proposals

[Laurent Godard]

Hi all

First, i must warn that i have little experience in c++ development; So 
my wording may be strange and inacurate. Feel free to request for 
rewording if something is not clear

manys thanks to all the devs who helped me on this (matus, mst)

i'll present in the comming line the approach i've tested regarding make 
perfcheck

The initial work has been pointing by Matus
The idea is to use valgrind while running tests
my starting point has been https://gerrit.libreoffice.org/#/c/11296

My goal was to enhance this with in mind that make perfcheck is only 
launched on demand as it is a costly process

Here are the adressed points; I present them in the order of which i 
developped them, may be inapropriate

1- percheck tests not restricted at class level
2- exploiting results
3- re use of existing tests for percheck

I'll point the gerrit commits in each of these points
They are first shoots and may need some (a lot of) polish
at least it builds and do what expected

Any comments welcome

my code is far from being perfect
let me know if it needs something to be done before being pushed to master

Thanks again to all of the dev that helped me


Laurent

==========================

1- percheck tests not restricted at class level
-----------------------------------------------

The problem being at classe level is that some tests may be very long 
and hide problems on other performed tests. Time can be doubled without 
noticing it

The goal here is to be able to DUMP callgrind each time we want it

Instead of using
CALLGRIND_DUMP_STATS

one can use
CALLGRIND_DUMP_STATS_AT(message);

--> this will create a callgrind.out file at each call
--> then we can separate the different part of a test
--> note the (message) that will be used in part 2

a first approach of a "big" test, with 3 dumps is here
https://gerrit.libreoffice.org/#/c/11949/

2- exploiting results
---------------------

The dumps are stored in workdir/CppunitTest/

Each dump file starting with callgrind.out contains
- the message of step 1
- the total of cycles

here is a python script that walk through workdir/CppunitTest/ and 
retreive these 2 information for each dump file
It also retreives the currently last commit

it append these results in a csv file
TODO: use this csv file to monitor or detect problems

https://gerrit.libreoffice.org/#/c/11962/

output example

$ cat perfcheckResult.csv
lastCommit      test name       filedatetime    dump comment    count
741c661ece19ccb4e94bb20ceb75d89a29b1b2a8        sc_perf_searchobj 
10/14/2014 09:54:52     testSheetFindAll - Search value 11403647297
741c661ece19ccb4e94bb20ceb75d89a29b1b2a8        sc_perf_searchobj 
10/14/2014 09:54:53     testSheetFindAll - Search style 767867
741c661ece19ccb4e94bb20ceb75d89a29b1b2a8        sc_perf_searchobj 
10/14/2014 09:52:06     Load Big File   17078422628

3- re use of existing tests for percheck
----------------------------------------

i tried this approach because monitoring perfs would lead to write 
duplicate tests

the naive question was : why not reuse existing tests that would start 
callgrind instrumentation only if we are in a perfcheck context ?

it would then be transparent for subsequentcheck and enhance perftest 
coverage

as we saw at step 1, one can start callgrind instrumentation for only 
few chosen lines

the basic idea, in an existing test, would be to write something like

startPerfInstrumentation();
   uno::Reference< container::XIndexAccess > xIndex = 
xSearchable->findAll(xSearchDescr);
   endPerfInstrumentation("testSheetFindAll - Search value");

where startPerfInstrumentation and endPerfInstrumentation do nothing if 
not in a perfcheck context

see whole code example at
https://gerrit.libreoffice.org/#/c/11982/

this context is set using$(eval $(call 
gb_CppunitTest_add_defs,sc_perf_searchobj,\
     -DIS_PERFCHECK \
))

see then
https://gerrit.libreoffice.org/#/c/11982/2/sc/qa/perf/perf_instrumentation.cxx,cm

the idea is to have 2 make files
- one for subsequent test
- one for perfcheck (that sets IS_PERFCHECK)

that point to the same source test but lets perf_instrumentation.cxx be 
re-build each time

globally seems to work with some tricky parts
(thanks mst for your help)
https://gerrit.libreoffice.org/#/c/11982/2/sc/qa/perf/scperfsearch.cxx,cm
i don't know if it can lead to some problems

ok, i stop here, ask me if something is not clear

------------------