[Piglit] [PATCH] framework/run.py: allow additional excluded tests on resume

[Mason, Michael W]

> -----Original Message-----
> From: Dylan Baker [mailto:baker.dylan.c at gmail.com]
> Sent: Tuesday, May 19, 2015 9:42 AM
> To: Daniel Vetter
> Cc: Mason, Michael W; piglit at lists.freedesktop.org
> Subject: Re: [Piglit] [PATCH] framework/run.py: allow additional excluded tests on resume
> 
> On Tue, May 19, 2015 at 12:06:54PM +0200, Daniel Vetter wrote:
> > On Mon, May 18, 2015 at 05:12:17PM -0700, Dylan Baker wrote:
> > > I'm not sure this is a good idea,
> > >
> > > Ken Graunke and I talked this to death when I did the refactor that
> > > split the run and resume code into separate paths, and we agreed
> > > that changing the tests during a resume would make it impossible to
> > > reproduce a run, since the regex you pass might exclude tests that
> > > have already completed.
> > >
> > > That said, I'm opened to being convinced it is a good idea.
> >
> > With your patches to mark a test as incomplete, should we instead just
> > have an option on resume to mark all incomplete tests as failed and
> > not try to restart them? Or is that already what happens (tbh I
> > haven't tried).
> > -Daniel
> 
> Currently they're always retried. It seems reasonable to me to either change the behavior or add a switch to change the
> behavior to either mark them as failed or just leave them as incomplete.

How about something like this instead of my original patch? I think we should preserve the fact that some tests didn't complete. I'll submit this in a separate email if you all agree.

--- a/framework/programs/run.py
+++ b/framework/programs/run.py
@@ -313,6 +313,9 @@ def resume(input_):
                         type=argparse.FileType("r"),
                         help="Optionally specify a piglit config file to use. "
                              "Default is piglit.conf")
+    parser.add_argument("-n", "--no-retry", dest='no_retry',
+                        action='store_true',
+                        help="Do not retry incomplete tests")
     args = parser.parse_args(input_)
     _disable_windows_exception_messages()
 
@@ -342,7 +345,7 @@ def resume(input_):
     # Don't re-run tests that have already completed, incomplete status tests
     # have obviously not completed.
     for name, result in results.tests.iteritems():
-        if result['result'] != 'incomplete':
+        if result['result'] != 'incomplete' or args.no_retry:
             opts.exclude_tests.add(name)


> 
> [snip]