[igt-dev] [PATCH i-g-t v2 4/5] runner: New test runner

[Petri Latvala]

On Thu, Jun 14, 2018 at 01:14:52PM -0700, Antonio Argenziano wrote:
> 
> 
> On 14/06/18 03:18, Petri Latvala wrote:
> > On Wed, Jun 13, 2018 at 10:10:11AM -0700, Antonio Argenziano wrote:
> > > 
> > > 
> > > On 13/06/18 03:07, Petri Latvala wrote:
> > > > This is a new test runner to replace piglit. Piglit has been very
> > > > useful as a test runner, but certain improvements have been very
> > > > difficult if possible at all in a generic test running framework.
> > > > 
> > > > Important improvements over piglit:
> > > > 
> > > > - Faster to launch. Being able to make assumptions about what we're
> > > >     executing makes it possible to save significant amounts of time. For
> > > >     example, a testlist file's line "igt at somebinary@somesubtest" already
> > > >     has all the information we need to construct the correct command
> > > >     line to execute that particular subtest, instead of listing all
> > > >     subtests of all test binaries and mapping them to command
> > > >     lines. Same goes for the regexp filters command line flags -t and
> > > >     -x; If we use -x somebinaryname, we don't need to list subtests from
> > > >     somebinaryname, we already know none of them will get executed.
> > > > 
> > > > - Logs of incomplete tests. Piglit collects test output to memory and
> > > >     dumps them to a file when the test is complete. The new runner
> > > >     writes all output to disk immediately.
> > > > 
> > > > - Ability to execute multiple subtests in one binary execution. This
> > > >     was possible with piglit, but its semantics made it very hard to
> > > >     implement in practice. For example, having a testlist file not only
> > > >     selected a subset of tests to run, but also mandated that they be
> > > >     executed in the same order.
> > > > 
> > > > - Flexible timeout support. Instead of mandating a time tests cannot
> > > >     exceed, the new runner has a timeout on inactivity. Activity is
> > > >     any output on the test's stdout or stderr, or kernel activity via
> > > >     /dev/kmsg.
> > > 
> > > Is there going to be also a (longer) timeout on execution time? It would
> > > help if a test is stuck in a loop (e.g. retrying on EAGAIN forever) and is
> > > therefore responsive but just not progressing forward.
> > 
> > What timeout values are used in CI and in which way is still up in the
> > air. With an inactivity timeout (as implemented here) we can have
> > individual tests that run for more than the 10 minutes that piglit
> > enforces (as long as they seem alive), and have timeouts earlier for
> > tests that don't seem active. Other possibilities (in addition to
> > inactivity timeout) are overall timeouts for "full execution of this
> > test list should take 15 minutes total, tops", or having a cap for an
> > individual subtest, an individual binary (all subtests it's going to
> > run in this execution), and combine those with expanding the testlist
> > format to specify the caps based on real data. Meaning,
> 
> I would prefer a large per-subtest execution timeout paired with a small
> inactivity timeout. Where the values of large and small would probably have
> to be chosen empirically. This way a 'bad' test would fail still but not
> affect the results from other tests in the same run.

My preference as well.


> 
> > 
> > igt at foobar@frob-some 60
> 
> Could this be auto-generated from somewhere?

Yeah, that's what I was trying to say. Those values would be generated
from real data from CI.


-- 
Petri Latvala