[igt-dev] [PATCH i-g-t v2 4/5] runner: New test runner

[Petri Latvala]

On Wed, Jun 13, 2018 at 10:10:11AM -0700, Antonio Argenziano wrote:
> 
> 
> On 13/06/18 03:07, Petri Latvala wrote:
> > This is a new test runner to replace piglit. Piglit has been very
> > useful as a test runner, but certain improvements have been very
> > difficult if possible at all in a generic test running framework.
> > 
> > Important improvements over piglit:
> > 
> > - Faster to launch. Being able to make assumptions about what we're
> >    executing makes it possible to save significant amounts of time. For
> >    example, a testlist file's line "igt at somebinary@somesubtest" already
> >    has all the information we need to construct the correct command
> >    line to execute that particular subtest, instead of listing all
> >    subtests of all test binaries and mapping them to command
> >    lines. Same goes for the regexp filters command line flags -t and
> >    -x; If we use -x somebinaryname, we don't need to list subtests from
> >    somebinaryname, we already know none of them will get executed.
> > 
> > - Logs of incomplete tests. Piglit collects test output to memory and
> >    dumps them to a file when the test is complete. The new runner
> >    writes all output to disk immediately.
> > 
> > - Ability to execute multiple subtests in one binary execution. This
> >    was possible with piglit, but its semantics made it very hard to
> >    implement in practice. For example, having a testlist file not only
> >    selected a subset of tests to run, but also mandated that they be
> >    executed in the same order.
> > 
> > - Flexible timeout support. Instead of mandating a time tests cannot
> >    exceed, the new runner has a timeout on inactivity. Activity is
> >    any output on the test's stdout or stderr, or kernel activity via
> >    /dev/kmsg.
> 
> Is there going to be also a (longer) timeout on execution time? It would
> help if a test is stuck in a loop (e.g. retrying on EAGAIN forever) and is
> therefore responsive but just not progressing forward.

What timeout values are used in CI and in which way is still up in the
air. With an inactivity timeout (as implemented here) we can have
individual tests that run for more than the 10 minutes that piglit
enforces (as long as they seem alive), and have timeouts earlier for
tests that don't seem active. Other possibilities (in addition to
inactivity timeout) are overall timeouts for "full execution of this
test list should take 15 minutes total, tops", or having a cap for an
individual subtest, an individual binary (all subtests it's going to
run in this execution), and combine those with expanding the testlist
format to specify the caps based on real data. Meaning,

igt at foobar@frob-some 60

would kill that process after 60 seconds.


-- 
Petri Latvala