[igt-dev] [PATCH i-g-t v2 4/5] runner: New test runner

[Antonio Argenziano]

On 14/06/18 03:18, Petri Latvala wrote:
> On Wed, Jun 13, 2018 at 10:10:11AM -0700, Antonio Argenziano wrote:
>>
>>
>> On 13/06/18 03:07, Petri Latvala wrote:
>>> This is a new test runner to replace piglit. Piglit has been very
>>> useful as a test runner, but certain improvements have been very
>>> difficult if possible at all in a generic test running framework.
>>>
>>> Important improvements over piglit:
>>>
>>> - Faster to launch. Being able to make assumptions about what we're
>>>     executing makes it possible to save significant amounts of time. For
>>>     example, a testlist file's line "igt at somebinary@somesubtest" already
>>>     has all the information we need to construct the correct command
>>>     line to execute that particular subtest, instead of listing all
>>>     subtests of all test binaries and mapping them to command
>>>     lines. Same goes for the regexp filters command line flags -t and
>>>     -x; If we use -x somebinaryname, we don't need to list subtests from
>>>     somebinaryname, we already know none of them will get executed.
>>>
>>> - Logs of incomplete tests. Piglit collects test output to memory and
>>>     dumps them to a file when the test is complete. The new runner
>>>     writes all output to disk immediately.
>>>
>>> - Ability to execute multiple subtests in one binary execution. This
>>>     was possible with piglit, but its semantics made it very hard to
>>>     implement in practice. For example, having a testlist file not only
>>>     selected a subset of tests to run, but also mandated that they be
>>>     executed in the same order.
>>>
>>> - Flexible timeout support. Instead of mandating a time tests cannot
>>>     exceed, the new runner has a timeout on inactivity. Activity is
>>>     any output on the test's stdout or stderr, or kernel activity via
>>>     /dev/kmsg.
>>
>> Is there going to be also a (longer) timeout on execution time? It would
>> help if a test is stuck in a loop (e.g. retrying on EAGAIN forever) and is
>> therefore responsive but just not progressing forward.
> 
> What timeout values are used in CI and in which way is still up in the
> air. With an inactivity timeout (as implemented here) we can have
> individual tests that run for more than the 10 minutes that piglit
> enforces (as long as they seem alive), and have timeouts earlier for
> tests that don't seem active. Other possibilities (in addition to
> inactivity timeout) are overall timeouts for "full execution of this
> test list should take 15 minutes total, tops", or having a cap for an
> individual subtest, an individual binary (all subtests it's going to
> run in this execution), and combine those with expanding the testlist
> format to specify the caps based on real data. Meaning,

I would prefer a large per-subtest execution timeout paired with a small 
inactivity timeout. Where the values of large and small would probably 
have to be chosen empirically. This way a 'bad' test would fail still 
but not affect the results from other tests in the same run.

> 
> igt at foobar@frob-some 60

Could this be auto-generated from somewhere?

Thanks,
Antonio

> 
> would kill that process after 60 seconds.
> 
>