[Piglit] [PATCH 0/5] Make piglit test ordering deterministic

[Brian Paul]

On 05/02/2016 03:21 PM, Dylan Baker wrote:
> This series makes the test ordering in piglit deterministic, in that it
> will run the tests in the order that they are provided by the profile.
> For native piglit tests (OpenGL and OpenCL) this will make them
> completely deterministic, for external integrated suits like dEQP and
> IGT it will be determined by the way the external suite enumerates it's
> tests; if the external tools is deterministic piglit will be too.
>
> An option for external suites that are not deterministic would be to
> implement a custom TestProfile class that sorts test TestDict using it's
> reorder() method.
>
> This further alters --test-list option to work with this internal change
> to impose ordering restrictions on the way tests are run.
>
> Brian, I hope this is what you're looking for.

Yes, it is.  Thanks!

I applied the patchs and tried a few test runs and it looks good.
I just did a light review of the code changes.

Acked-by: Brian Paul <brianp at vmware.com>
Tested-by: Brian Paul <brianp at vmware.com>

One other issue though.  It would be nice if the output of 
piglit-print-commands.py could be used as a test list input to 
piglit-run.py  When I tried it, piglit-run.py spun its wheels for a 
while then hit an exception.

-Brian


>
> Daniela, I don't know exactly what you were looking for when you asked
> about sort order, but I thought I would point you at this, since Brian
> asked for something (that I think) is similar to what you're looking
> for. (https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.freedesktop.org_show-5Fbug.cgi-3Fid-3D95053&d=BQIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=T0t4QG7chq2ZwJo6wilkFznRSFy-8uDKartPGbomVj8&m=P0idU-rSN_1ROsAgtRw0-jqXAQvrw1lv886Sa7DT818&s=KhSdC9bxSuoRx9psA_FmzE8UIKCcTtyWCzT7zBXOQ08&e= )
>
> Dylan Baker (5):
>    framework/profile: Convert TestDict to a collections.MutableMapping
>    framework/profile: don't replace TestDict instance with regular dict
>    framework/profile: Make test run order deterministic
>    framework: Make JSON backend test ordering deterministic
>    framework: Make testlist option impose ordering
>
>   framework/backends/json.py |  9 ++---
>   framework/profile.py       | 84 +++++++++++++++++++++++++++++++++++-----------
>   framework/programs/run.py  | 12 +++----
>   framework/results.py       |  2 +-
>   unittests/profile_tests.py | 55 +++++++++++++++++++-----------
>   5 files changed, 112 insertions(+), 50 deletions(-)
>