[poppler] [RFC] Extend regtest framework to track performance

[Albert Astals Cid]

El Wednesday 30 December 2015, a les 17:04:42, Adam Reichold va escriure:
> Hello again,
> 
> as discussed in the code modernization thread, if we are going to make
> performance-orient changes, we need a simple way to track functional and
> performance regressions.
> 
> The attached patch tries to extend the existing Python-based regtest
> framework to measure run time and memory usage to spot significant
> performance changes in the sense of relative deviations w.r.t. to these
> two parameters. It also collects the sums of both which might be used as
> "ball park" numbers to compare the performance effect of changes over
> document collections.

Have you tried it? How stable are the numbers? For example here i get for 
rendering the same file (discarding the first time that is loading the file 
into memory) numbers that range from 620ms to 676ms, i.e. ~10% variation 
without no change at all.

Cheers,
  Albert

> 
> The patch runs the measured commands repeatedly including warm-up
> iterations and collects statistics from these runs. The measurement
> results are stored as JSON documents with the actual program output of
> e.g. pdftotext or pdftoppm being discarded.
> 
> To implement the check for relative deviations, it abuses the checksum
> comparison method and hence checksums are still computed for the JSON
> documents even though they are actually unnecessary. It is also limited
> to Unix-like operating systems (due to the use of the wait3 syscall to
> determine resource usage similar to the time command).