[Piglit] [PATCH] framework/run.py: allow additional excluded tests on resume

[Mason, Michael W]

> -----Original Message-----
> From: Daniel Vetter [mailto:daniel.vetter at ffwll.ch] On Behalf Of Daniel Vetter
> Sent: Tuesday, May 19, 2015 3:07 AM
> To: Dylan Baker
> Cc: Mason, Michael W; piglit at lists.freedesktop.org
> Subject: Re: [Piglit] [PATCH] framework/run.py: allow additional excluded tests on resume
> 
> On Mon, May 18, 2015 at 05:12:17PM -0700, Dylan Baker wrote:
> > I'm not sure this is a good idea,
> >
> > Ken Graunke and I talked this to death when I did the refactor that
> > split the run and resume code into separate paths, and we agreed that
> > changing the tests during a resume would make it impossible to
> > reproduce a run, since the regex you pass might exclude tests that
> > have already completed.
> >
> > That said, I'm opened to being convinced it is a good idea.
> 
> With your patches to mark a test as incomplete, should we instead just have an option on resume to mark all incomplete
> tests as failed and not try to restart them? Or is that already what happens (tbh I haven't tried).
> -Daniel

I haven't tried either, but the code suggests incomplete tests are always retried on resume.  Are you recommending an option that changes the logged result from 'incomplete' to 'failed' or that simply doesn't retry incomplete tests.  I vote for the latter. That would preserve the list of incomplete tests and accomplish the intent, if not the effect, of my patch.

> 
> >
> > Dylan
> >
> > On Mon, May 18, 2015 at 02:29:58PM -0700, Mike Mason wrote:
> > > This patch allows additional tests to be excluded when resuming a
> > > test run. This is especially useful when a failing test causes a
> > > crash or reboot. Currently, that same test runs again when
> > > attempting to resume the test run, resulting in the same crash or
> > > reboot.
> > >
> > > Note that this does not change the exclude options saved at the
> > > start of the run. If a run is resumed multiple times, any additional
> > > excluded tests must be specified on the command line each time.
> > >
> > > Signed-off-by: Mike Mason <michael.w.mason at intel.com>
> > > ---
> > >  framework/programs/run.py | 14 +++++++++++++-
> > >  1 file changed, 13 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/framework/programs/run.py b/framework/programs/run.py
> > > index 6053074..89e4b69 100644
> > > --- a/framework/programs/run.py
> > > +++ b/framework/programs/run.py
> > > @@ -308,6 +308,12 @@ def resume(input_):
> > >                          type=path.realpath,
> > >                          metavar="<Results Path>",
> > >                          help="Path to results folder")
> > > +    parser.add_argument("-x", "--exclude-tests",
> > > +                        default=[],
> > > +                        action="append",
> > > +                        metavar="<regex>",
> > > +                        help="Add tests to exclude "
> > > +                             "(can be used more than once)")
> > >      parser.add_argument("-f", "--config",
> > >                          dest="config_file",
> > >                          type=argparse.FileType("r"), @@ -317,8
> > > +323,14 @@ def resume(input_):
> > >      _disable_windows_exception_messages()
> > >
> > >      results = backends.load(args.results_path)
> > > +
> > > +    # Additional tests to exclude from resumed run
> > > +    exclude_filter = results.options['exclude_filter']
> > > +    if args.exclude_tests:
> > > +        exclude_filter.extend(args.exclude_tests)
> > > +
> > >      opts = core.Options(concurrent=results.options['concurrent'],
> > > -                        exclude_filter=results.options['exclude_filter'],
> > > +                        exclude_filter=exclude_filter,
> > >                          include_filter=results.options['filter'],
> > >                          execute=results.options['execute'],
> > >                          valgrind=results.options['valgrind'],
> > > --
> > > 2.1.0
> > >
> > > _______________________________________________
> > > Piglit mailing list
> > > Piglit at lists.freedesktop.org
> > > http://lists.freedesktop.org/mailman/listinfo/piglit
> 
> 
> 
> > _______________________________________________
> > Piglit mailing list
> > Piglit at lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/piglit
> 
> 
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch