[Piglit] [PATCH 0/5] Make piglit test ordering deterministic

[Dylan Baker]

Right. So without the patches the tests are sorted by some implementation detail of Python, with these patches they are guaranteed to be ordered the same as they are given in the profile.

Dylan

On May 3, 2016 8:29:13 AM PDT, Marius Vlad <marius.c.vlad at intel.com> wrote:
>So, with these patches we won't have that ``randomized'' feature at all
>right? (unless the test list is given as such).
>
>On Mon, May 02, 2016 at 02:21:09PM -0700, Dylan Baker wrote:
>> This series makes the test ordering in piglit deterministic, in that
>it
>> will run the tests in the order that they are provided by the
>profile.
>> For native piglit tests (OpenGL and OpenCL) this will make them
>> completely deterministic, for external integrated suits like dEQP and
>> IGT it will be determined by the way the external suite enumerates
>it's
>> tests; if the external tools is deterministic piglit will be too.
>> 
>> An option for external suites that are not deterministic would be to
>> implement a custom TestProfile class that sorts test TestDict using
>it's
>> reorder() method.
>> 
>> This further alters --test-list option to work with this internal
>change
>> to impose ordering restrictions on the way tests are run.
>> 
>> Brian, I hope this is what you're looking for.
>> 
>> Daniela, I don't know exactly what you were looking for when you
>asked
>> about sort order, but I thought I would point you at this, since
>Brian
>> asked for something (that I think) is similar to what you're looking
>> for. (https://bugs.freedesktop.org/show_bug.cgi?id=95053)
>> 
>> Dylan Baker (5):
>>   framework/profile: Convert TestDict to a collections.MutableMapping
>>   framework/profile: don't replace TestDict instance with regular
>dict
>>   framework/profile: Make test run order deterministic
>>   framework: Make JSON backend test ordering deterministic
>>   framework: Make testlist option impose ordering
>> 
>>  framework/backends/json.py |  9 ++---
>>  framework/profile.py       | 84
>+++++++++++++++++++++++++++++++++++-----------
>>  framework/programs/run.py  | 12 +++----
>>  framework/results.py       |  2 +-
>>  unittests/profile_tests.py | 55 +++++++++++++++++++-----------
>>  5 files changed, 112 insertions(+), 50 deletions(-)
>> 
>> -- 
>> 2.8.2
>> 
>> _______________________________________________
>> Piglit mailing list
>> Piglit at lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/piglit

Sent from my phone
please excuse spelling problems
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/piglit/attachments/20160503/07ce5afd/attachment-0001.html>