[Piglit] [PATCH] framework: add a timeout command line parameter

[Dylan Baker]

On Mon, Jul 13, 2015 at 02:09:26PM +0100, Thomas Wood wrote:
> On 10 July 2015 at 19:24, Dylan Baker <baker.dylan.c at gmail.com> wrote:
> 
> > Hi Thomas,
> >
> > I meant to get back to you earlier on this, but I forgot yesterday when
> > I got bogged down in Jenkins madness.
> >
> > Anyway, if you have a look at the (I think) 2nd patch in my python3
> > series, you'll notice that I had the idea that we could set timeouts in
> > classes on a per class basis. This presents a problem for using a
> > universal test time overwrite, especially when you consider that we also
> > may need per-test overwrites (some of the piglit tests are really long,
> > some of them are really short).
> >
> > Do you want to have an igt only override, or should we try to figure out
> > how to make something that will work universally?
> >
> 
> The idea was to allow the user to temporarily change the timeout that
> applied to all tests. It could include a parameter to ensure that timeout
> durations are only ever reduced or increased.
> 

Okay, well here's the real problem I see. I designed the timeouts such
that the class sets a sane timeout for the test class in general, and
then tests that fall outside that norm can override. So the
implementation looks something like:

```python
class BaseTest(object):
    timeout = 50

class Test(BaseTest):
    timeout = 100

class Test2(BaseTest):
    pass

foo = Test2()
foo.timeout = 400
```

Now the problem with this plus your model is that either:
we override the BaseTest class timeout then any test the defines a
timeout parameter (like Test) will not receive that override from
BaseTest, nor will the foo instance, since it sets its own timeout.

In python3 I think we can solve this via a class decorator, but for
python2 that's a much harder proposition to fix. Although maybe it
doesn't matter as much in python2, since timeouts are not thread safe,
not portable, and are only implemented by igt in python2

But I'd really like to keep timeouts a property of each test class,
because reasonable timeout durations really are not universal. 120
seconds is pretty reasonable for glean, 30 for most piglit tests, 10 for
glsl parser tests, and 600 for igt tests, but we don't want to waith 600
seconds for a glsl parser test to return, if it doesn't return after 10
seconds something is wrong.

Dylan

[snip]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: Digital signature
URL: <http://lists.freedesktop.org/archives/piglit/attachments/20150713/6c1876d3/attachment.sig>