[Piglit] [PATCH] framework: add a timeout command line parameter

[Daniel Vetter]

On Mon, Jul 20, 2015 at 01:57:11PM +0100, Thomas Wood wrote:
> On 13 July 2015 at 23:09, Dylan Baker <baker.dylan.c at gmail.com> wrote:
> >
> > On Mon, Jul 13, 2015 at 02:09:26PM +0100, Thomas Wood wrote:
> > > On 10 July 2015 at 19:24, Dylan Baker <baker.dylan.c at gmail.com> wrote:
> > >
> > > > Hi Thomas,
> > > >
> > > > I meant to get back to you earlier on this, but I forgot yesterday when
> > > > I got bogged down in Jenkins madness.
> > > >
> > > > Anyway, if you have a look at the (I think) 2nd patch in my python3
> > > > series, you'll notice that I had the idea that we could set timeouts in
> > > > classes on a per class basis. This presents a problem for using a
> > > > universal test time overwrite, especially when you consider that we also
> > > > may need per-test overwrites (some of the piglit tests are really long,
> > > > some of them are really short).
> > > >
> > > > Do you want to have an igt only override, or should we try to figure out
> > > > how to make something that will work universally?
> > > >
> > >
> > > The idea was to allow the user to temporarily change the timeout that
> > > applied to all tests. It could include a parameter to ensure that timeout
> > > durations are only ever reduced or increased.
> > >
> >
> > Okay, well here's the real problem I see. I designed the timeouts such
> > that the class sets a sane timeout for the test class in general, and
> > then tests that fall outside that norm can override. So the
> > implementation looks something like:
> >
> > ```python
> > class BaseTest(object):
> >     timeout = 50
> >
> > class Test(BaseTest):
> >     timeout = 100
> >
> > class Test2(BaseTest):
> >     pass
> >
> > foo = Test2()
> > foo.timeout = 400
> > ```
> >
> > Now the problem with this plus your model is that either:
> > we override the BaseTest class timeout then any test the defines a
> > timeout parameter (like Test) will not receive that override from
> > BaseTest, nor will the foo instance, since it sets its own timeout.
> 
> 
> 
> Could the subclasses check the timeout specified by the user and
> decide whether or not to apply it?

What about just adding a --max-timeout and --min-timeout for the cmdline?
I think that should cover all uses-cases we have in mind really. Then we
could use that to adjust the per-testcaes timeout in a suitable way.
-Daniel

> 
> 
> >
> >
> > In python3 I think we can solve this via a class decorator, but for
> > python2 that's a much harder proposition to fix. Although maybe it
> > doesn't matter as much in python2, since timeouts are not thread safe,
> > not portable, and are only implemented by igt in python2
> >
> > But I'd really like to keep timeouts a property of each test class,
> > because reasonable timeout durations really are not universal. 120
> > seconds is pretty reasonable for glean, 30 for most piglit tests, 10 for
> > glsl parser tests, and 600 for igt tests, but we don't want to waith 600
> > seconds for a glsl parser test to return, if it doesn't return after 10
> > seconds something is wrong.
> >
> > Dylan
> >
> > [snip]
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/piglit

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch