[igt-dev] [PATCH i-g-t v2 4/5] runner: New test runner

[Joonas Lahtinen]

Quoting Petri Latvala (2018-06-15 11:53:05)
> On Thu, Jun 14, 2018 at 01:14:52PM -0700, Antonio Argenziano wrote:
> > 
> > 
> > On 14/06/18 03:18, Petri Latvala wrote:
> > > On Wed, Jun 13, 2018 at 10:10:11AM -0700, Antonio Argenziano wrote:
> > > > 
> > > > 
> > > > On 13/06/18 03:07, Petri Latvala wrote:
> > > > > This is a new test runner to replace piglit. Piglit has been very
> > > > > useful as a test runner, but certain improvements have been very
> > > > > difficult if possible at all in a generic test running framework.
> > > > > 
> > > > > Important improvements over piglit:
> > > > > 
> > > > > - Faster to launch. Being able to make assumptions about what we're
> > > > >     executing makes it possible to save significant amounts of time. For
> > > > >     example, a testlist file's line "igt at somebinary@somesubtest" already
> > > > >     has all the information we need to construct the correct command
> > > > >     line to execute that particular subtest, instead of listing all
> > > > >     subtests of all test binaries and mapping them to command
> > > > >     lines. Same goes for the regexp filters command line flags -t and
> > > > >     -x; If we use -x somebinaryname, we don't need to list subtests from
> > > > >     somebinaryname, we already know none of them will get executed.
> > > > > 
> > > > > - Logs of incomplete tests. Piglit collects test output to memory and
> > > > >     dumps them to a file when the test is complete. The new runner
> > > > >     writes all output to disk immediately.
> > > > > 
> > > > > - Ability to execute multiple subtests in one binary execution. This
> > > > >     was possible with piglit, but its semantics made it very hard to
> > > > >     implement in practice. For example, having a testlist file not only
> > > > >     selected a subset of tests to run, but also mandated that they be
> > > > >     executed in the same order.
> > > > > 
> > > > > - Flexible timeout support. Instead of mandating a time tests cannot
> > > > >     exceed, the new runner has a timeout on inactivity. Activity is
> > > > >     any output on the test's stdout or stderr, or kernel activity via
> > > > >     /dev/kmsg.
> > > > 
> > > > Is there going to be also a (longer) timeout on execution time? It would
> > > > help if a test is stuck in a loop (e.g. retrying on EAGAIN forever) and is
> > > > therefore responsive but just not progressing forward.
> > > 
> > > What timeout values are used in CI and in which way is still up in the
> > > air. With an inactivity timeout (as implemented here) we can have
> > > individual tests that run for more than the 10 minutes that piglit
> > > enforces (as long as they seem alive), and have timeouts earlier for
> > > tests that don't seem active. Other possibilities (in addition to
> > > inactivity timeout) are overall timeouts for "full execution of this
> > > test list should take 15 minutes total, tops", or having a cap for an
> > > individual subtest, an individual binary (all subtests it's going to
> > > run in this execution), and combine those with expanding the testlist
> > > format to specify the caps based on real data. Meaning,
> > 
> > I would prefer a large per-subtest execution timeout paired with a small
> > inactivity timeout. Where the values of large and small would probably have
> > to be chosen empirically. This way a 'bad' test would fail still but not
> > affect the results from other tests in the same run.
> 
> My preference as well.
> 
> 
> > 
> > > 
> > > igt at foobar@frob-some 60

This parameter then could be passed within a variable to the test. On
the GEM side we have a lot of tests that can be budgeted based on
available time.

Although, that comes with the caveat of understanding that passing such
a test makes the validity of the test depend on the time it's allocated.

So we should maybe track the cumulated runtimes of subtests per machine
after such a change, so that we can make some statistical analysis.

Regards, Joonas

> > 
> > Could this be auto-generated from somewhere?
> 
> Yeah, that's what I was trying to say. Those values would be generated
> from real data from CI.
> 
> 
> -- 
> Petri Latvala
> _______________________________________________
> igt-dev mailing list
> igt-dev at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/igt-dev