<html><head></head><body>Right. So without the patches the tests are sorted by some implementation detail of Python, with these patches they are guaranteed to be ordered the same as they are given in the profile. Dylan <div class="gmail_quote">On May 3, 2016 8:29:13 AM PDT, Marius Vlad <marius.c.vlad@intel.com> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"> <pre class="k9mail">So, with these patches we won't have that ``randomized'' feature at all right? (unless the test list is given as such). On Mon, May 02, 2016 at 02:21:09PM -0700, Dylan Baker wrote: <blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #729fcf; padding-left: 1ex;"> This series makes the test ordering in piglit deterministic, in that it will run the tests in the order that they are provided by the profile. For native piglit tests (OpenGL and OpenCL) this will make them completely deterministic, for external integrated suits like dEQP and IGT it will be determined by the way the external suite enumerates it's tests; if the external tools is deterministic piglit will be too. An option for external suites that are not deterministic would be to implement a custom TestProfile class that sorts test TestDict using it's reorder() method. This further alters --test-list option to work with this internal change to impose ordering restrictions on the way tests are run. Brian, I hope this is what you're looking for. Daniela, I don't know exactly what you were looking for when you asked about sort order, but I thought I would point you at this, since Brian asked for something (that I think) is similar to what you're looking for. (<a href="https://bugs.freedesktop.org/show_bug.cgi?id=95053">https://bugs.freedesktop.org/show_bug.cgi?id=95053</a>) Dylan Baker (5): framework/profile: Convert TestDict to a collections.MutableMapping framework/profile: don't replace TestDict instance with regular dict framework/profile: Make test run order deterministic framework: Make JSON backend test ordering deterministic framework: Make testlist option impose ordering framework/backends/<a href="http://json.py">json.py</a> | 9 ++--- framework/<a href="http://profile.py">profile.py</a> | 84 +++++++++++++++++++++++++++++++++++----------- framework/programs/<a href="http://run.py">run.py</a> | 12 +++---- framework/<a href="http://results.py">results.py</a> | 2 +- unittests/profile_<a href="http://tests.py">tests.py</a> | 55 +++++++++++++++++++----------- 5 files changed, 112 insertions(+), 50 deletions(-) -- 2.8.2 <hr /> Piglit mailing list Piglit@lists.freedesktop.org <a href="https://lists.freedesktop.org/mailman/listinfo/piglit">https://lists.freedesktop.org/mailman/listinfo/piglit</a> </blockquote></pre></blockquote></div> Sent from my phone please excuse spelling problems</body></html>