[poppler] [RFC] Extend regtest framework to track performance

[Adam Reichold]

Hello again,

Am 31.12.2015 um 01:11 schrieb Albert Astals Cid:
> El Thursday 31 December 2015, a les 00:58:10, Adam Reichold va escriure:
>> Hello again,
>>
>> Am 31.12.2015 um 00:21 schrieb Albert Astals Cid:
>>> El Wednesday 30 December 2015, a les 20:00:17, Ihar Filipau va escriure:
>>>> On 12/30/15, Albert Astals Cid <aacid at kde.org> wrote:
>>>>> El Wednesday 30 December 2015, a les 17:04:42, Adam Reichold va 
> escriure:
>>>>>> Hello again,
>>>>>>
>>>>>> as discussed in the code modernization thread, if we are going to make
>>>>>> performance-orient changes, we need a simple way to track functional
>>>>>> and
>>>>>> performance regressions.
>>>>>>
>>>>>> The attached patch tries to extend the existing Python-based regtest
>>>>>> framework to measure run time and memory usage to spot significant
>>>>>> performance changes in the sense of relative deviations w.r.t. to these
>>>>>> two parameters. It also collects the sums of both which might be used
>>>>>> as
>>>>>> "ball park" numbers to compare the performance effect of changes over
>>>>>> document collections.
>>>>>
>>>>> Have you tried it? How stable are the numbers? For example here i get
>>>>> for
>>>>> rendering the same file (discarding the first time that is loading the
>>>>> file
>>>>> into memory) numbers that range from 620ms to 676ms, i.e. ~10% variation
>>>>> without no change at all.
>>
>> Do you refer to the numbers provided by the patch or to manually running
>> e.g. pdftoppm? If you refer to the patch, which iteration counts did you
>> use?
> 
> Just test-poppler-qt4

I think this is to be expected and the reason why the patch uses a
configurable number of warm-up and measurement iterations. The actual
value of these parameters is probably something that needs to be
determined by experimentation, but AFAIK repeated measurements should
handle this kind of variation.

Best regards, Adam.

> _______________________________________________
> poppler mailing list
> poppler at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/poppler
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freedesktop.org/archives/poppler/attachments/20151231/9bbd4790/attachment.sig>