[poppler] [RFC] Extend regtest framework to track performance

[Adam Reichold]

Hello again,

attached is an extended version of the patch series that does better
handling of documents that fail to produce an output, reduces time spent
in console IO by discarding stdout and stderr and reduces boiler plate
code by using a common perf backend base class. (The patch series is
larger, but the final diff w.r.t. master is smaller. Should I rather
squash for a shorter patch series instead of the full commit history?)

Best regards, Adam.

Am 30.12.2015 um 17:04 schrieb Adam Reichold:
> Hello again,
> 
> as discussed in the code modernization thread, if we are going to make
> performance-orient changes, we need a simple way to track functional and
> performance regressions.
> 
> The attached patch tries to extend the existing Python-based regtest
> framework to measure run time and memory usage to spot significant
> performance changes in the sense of relative deviations w.r.t. to these
> two parameters. It also collects the sums of both which might be used as
> "ball park" numbers to compare the performance effect of changes over
> document collections.
> 
> The patch runs the measured commands repeatedly including warm-up
> iterations and collects statistics from these runs. The measurement
> results are stored as JSON documents with the actual program output of
> e.g. pdftotext or pdftoppm being discarded.
> 
> To implement the check for relative deviations, it abuses the checksum
> comparison method and hence checksums are still computed for the JSON
> documents even though they are actually unnecessary. It is also limited
> to Unix-like operating systems (due to the use of the wait3 syscall to
> determine resource usage similar to the time command).
> 
> 
> 
> _______________________________________________
> poppler mailing list
> poppler at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/poppler
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: regtest-performance-v2.patch
Type: text/x-patch
Size: 49918 bytes
Desc: not available
URL: <http://lists.freedesktop.org/archives/poppler/attachments/20151231/3359b641/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freedesktop.org/archives/poppler/attachments/20151231/3359b641/attachment-0001.sig>