[igt-dev] [PATCH i-g-t v2 4/5] runner: New test runner

[Antonio Argenziano]

On 13/06/18 03:07, Petri Latvala wrote:
> This is a new test runner to replace piglit. Piglit has been very
> useful as a test runner, but certain improvements have been very
> difficult if possible at all in a generic test running framework.
> 
> Important improvements over piglit:
> 
> - Faster to launch. Being able to make assumptions about what we're
>    executing makes it possible to save significant amounts of time. For
>    example, a testlist file's line "igt at somebinary@somesubtest" already
>    has all the information we need to construct the correct command
>    line to execute that particular subtest, instead of listing all
>    subtests of all test binaries and mapping them to command
>    lines. Same goes for the regexp filters command line flags -t and
>    -x; If we use -x somebinaryname, we don't need to list subtests from
>    somebinaryname, we already know none of them will get executed.
> 
> - Logs of incomplete tests. Piglit collects test output to memory and
>    dumps them to a file when the test is complete. The new runner
>    writes all output to disk immediately.
> 
> - Ability to execute multiple subtests in one binary execution. This
>    was possible with piglit, but its semantics made it very hard to
>    implement in practice. For example, having a testlist file not only
>    selected a subset of tests to run, but also mandated that they be
>    executed in the same order.
> 
> - Flexible timeout support. Instead of mandating a time tests cannot
>    exceed, the new runner has a timeout on inactivity. Activity is
>    any output on the test's stdout or stderr, or kernel activity via
>    /dev/kmsg.

Is there going to be also a (longer) timeout on execution time? It would 
help if a test is stuck in a loop (e.g. retrying on EAGAIN forever) and 
is therefore responsive but just not progressing forward.

Thanks,
Antonio