[Piglit] [PATCH v3] framework: Add --ignore-missing option

[Arkadiusz Hiler]

On Mon, Aug 14, 2017 at 10:21:13AM -0700, Dylan Baker wrote:
> Quoting Arkadiusz Hiler (2017-08-14 05:09:01)
> > Currently, if a test from provided testlist fails to be discovered by
> > the framework, piglit blows up with an exception.
> > 
> > This is both good - for consistency/early errors - and bad - for
> > handling some CI/automation scenarios (e.g autobisecting the tests).
> > 
> > So let's keep the current default, but allow some flexibility with the
> > new option that reports the missing tests as 'notrun'.
> > 
> > v2:
> >  - change switch name to 'ignore', as skip is too suggestive
> >  - use DummyTest to get 'notrun' result instead of warnings
> > 
> > v3: don't use OPTIONS
> > 
> > bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=99649
> > Cc: Dylan Baker <dylan at pnwbakers.com>
> > Cc: Tomi Sarvela <tomi.p.sarvela at intel.com>
> > Cc: Martin Peres <martin.peres at intel.com>
> > Signed-off-by: Arkadiusz Hiler <arkadiusz.hiler at intel.com>
> > ---
> >  framework/profile.py      | 10 +++++++---
> >  framework/programs/run.py | 12 ++++++++++++
> >  framework/test/base.py    | 12 ++++++++++++
> >  3 files changed, 31 insertions(+), 3 deletions(-)
> > 
> > diff --git a/framework/profile.py b/framework/profile.py
> > index a625318..cf0e298 100644
<SNIP>
> > @@ -314,7 +315,10 @@ class TestProfile(object):
> >          if self.forced_test_list:
> >              opts = collections.OrderedDict()
> >              for n in self.forced_test_list:
> > -                opts[n] = self.test_list[n]
> > +                if self.options['ignore_missing'] and n not in self.test_list:
> > +                    opts[n] = DummyTest(name=n, result=status.NOTRUN)
> 
> name and result are not keyword arguments, they're positional, so please write
> it like this instead:
> opts[n] = DummyTest(n, status.NOTRUN)
>
> As long as this passes the unit test suite with the above changed,
> Reviewed-by: Dylan Baker <dylan at pnwbakers.com>

% tox > with_patches
% git checkout HEAD~
% tox > without_patches
% diff -u <(grep -v PYTHONHASHSEED with_patches) <(grep -v PYTHONHASHSEED without_patches) | sprunge
http://sprunge.us/VZDN


Exactly identical pass rate and the same tests are failing.
I guess that's a pass? :-)

> Thanks for fixing that for me.

Glad to see you back in action.

v4 with your r-b is comming in a second and thanks for the review!


-- 
Cheers,
Arek