[Piglit] [PATCH] framework: add a timeout command line parameter

[Thomas Wood]

On 13 July 2015 at 23:09, Dylan Baker <baker.dylan.c at gmail.com> wrote:
>
> On Mon, Jul 13, 2015 at 02:09:26PM +0100, Thomas Wood wrote:
> > On 10 July 2015 at 19:24, Dylan Baker <baker.dylan.c at gmail.com> wrote:
> >
> > > Hi Thomas,
> > >
> > > I meant to get back to you earlier on this, but I forgot yesterday when
> > > I got bogged down in Jenkins madness.
> > >
> > > Anyway, if you have a look at the (I think) 2nd patch in my python3
> > > series, you'll notice that I had the idea that we could set timeouts in
> > > classes on a per class basis. This presents a problem for using a
> > > universal test time overwrite, especially when you consider that we also
> > > may need per-test overwrites (some of the piglit tests are really long,
> > > some of them are really short).
> > >
> > > Do you want to have an igt only override, or should we try to figure out
> > > how to make something that will work universally?
> > >
> >
> > The idea was to allow the user to temporarily change the timeout that
> > applied to all tests. It could include a parameter to ensure that timeout
> > durations are only ever reduced or increased.
> >
>
> Okay, well here's the real problem I see. I designed the timeouts such
> that the class sets a sane timeout for the test class in general, and
> then tests that fall outside that norm can override. So the
> implementation looks something like:
>
> ```python
> class BaseTest(object):
>     timeout = 50
>
> class Test(BaseTest):
>     timeout = 100
>
> class Test2(BaseTest):
>     pass
>
> foo = Test2()
> foo.timeout = 400
> ```
>
> Now the problem with this plus your model is that either:
> we override the BaseTest class timeout then any test the defines a
> timeout parameter (like Test) will not receive that override from
> BaseTest, nor will the foo instance, since it sets its own timeout.



Could the subclasses check the timeout specified by the user and
decide whether or not to apply it?


>
>
> In python3 I think we can solve this via a class decorator, but for
> python2 that's a much harder proposition to fix. Although maybe it
> doesn't matter as much in python2, since timeouts are not thread safe,
> not portable, and are only implemented by igt in python2
>
> But I'd really like to keep timeouts a property of each test class,
> because reasonable timeout durations really are not universal. 120
> seconds is pretty reasonable for glean, 30 for most piglit tests, 10 for
> glsl parser tests, and 600 for igt tests, but we don't want to waith 600
> seconds for a glsl parser test to return, if it doesn't return after 10
> seconds something is wrong.
>
> Dylan
>
> [snip]