[poppler] [RFC] Extend regtest framework to track performance

[Carlos Garcia Campos]

Albert Astals Cid <aacid at kde.org> writes:

> El Wednesday 30 December 2015, a les 17:04:42, Adam Reichold va escriure:
>> Hello again,
>> 
>> as discussed in the code modernization thread, if we are going to make
>> performance-orient changes, we need a simple way to track functional and
>> performance regressions.
>> 
>> The attached patch tries to extend the existing Python-based regtest
>> framework to measure run time and memory usage to spot significant
>> performance changes in the sense of relative deviations w.r.t. to these
>> two parameters. It also collects the sums of both which might be used as
>> "ball park" numbers to compare the performance effect of changes over
>> document collections.
>
> Have you tried it? How stable are the numbers? For example here i get for 
> rendering the same file (discarding the first time that is loading the file 
> into memory) numbers that range from 620ms to 676ms, i.e. ~10% variation 
> without no change at all.

I haven't looked at the patches in detail yet, but I don't think the
regtest framework should be the one measuring. I would add a tool for
that, pdfperf or something like that, that could measure the internals
(parsing, output devs, etc.) in a more detailed way. If we
need to get more information from the poppler core we could just add a
compile option to provide that info. And the regtest framework should
just run the perf command and collect the results. A report command
could compare results with refs or previous executions. I also think
performance tests should be run with a different command, since we don't
want to measure the performance of every single document we have in our
test suite (it would take too long). Instead, we could select a set of
documents with different kind of pdf features to run perf tests on those
only.

So, in summary, I would use a dedicated tool (depending on a build
option if we need to get more info from the core), and maybe even a
dedicated perf framework on top of that tool, since I consider perf tests
different from rendering regression tests, the same way unit tests are
handled by a different framework too.

> Cheers,
>   Albert
>
>> 
>> The patch runs the measured commands repeatedly including warm-up
>> iterations and collects statistics from these runs. The measurement
>> results are stored as JSON documents with the actual program output of
>> e.g. pdftotext or pdftoppm being discarded.
>> 
>> To implement the check for relative deviations, it abuses the checksum
>> comparison method and hence checksums are still computed for the JSON
>> documents even though they are actually unnecessary. It is also limited
>> to Unix-like operating systems (due to the use of the wait3 syscall to
>> determine resource usage similar to the time command).
>
> _______________________________________________
> poppler mailing list
> poppler at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/poppler

-- 
Carlos Garcia Campos
PGP key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x523E6462
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 180 bytes
Desc: not available
URL: <http://lists.freedesktop.org/archives/poppler/attachments/20151230/e71533d3/attachment.sig>