[Intel-gfx] [PATCH i-g-t 2/3] Unify handling of slow/combinatorial tests
Paulo Zanoni
przanoni at gmail.com
Fri Oct 30 04:55:03 PDT 2015
2015-10-30 5:56 GMT-02:00 David Weinehall <david.weinehall at linux.intel.com>:
> On Wed, Oct 28, 2015 at 02:12:15PM -0200, Paulo Zanoni wrote:
>> 2015-10-28 9:29 GMT-02:00 David Weinehall <david.weinehall at linux.intel.com>:
>> > Some tests should not be run by default, due to their slow,
>> > and sometimes superfluous, nature.
>> >
>> > We still want to be able to run these tests in some cases.
>> > Until now there's been no unified way of handling this. Remedy
>> > this by introducing the --all option to igt_core,
>> > and use it in gem_concurrent_blit & kms_frontbuffer_tracking.
>>
>> I really think you should explain both your plan and its
>> implementation in more details here.
>
> Well, I don't see how much more there is to explain; the idea is simply
> that different tests shouldn't implement similar behaviour in different
> manners (current kms_frontbuffer_tracking uses a command line option,
> gem_concurrent_blit changes behaviour depending on the file name it's
> called with).
What made me write that is that I noticed that now --all is required
during --list-subtests (thanks for doing this!) but it was not easy to
notice this in the commit message or in the code. So I was thinking
something simple, such as a description of how to use the new option
both when running IGT and when writing subtests:
"These tests will only appear in --list-subtests if you also specify
--all. Same for --run-subtest calls. There's this new macro
igt_subtest_slow_f (maybe igt_subtest_flags_f now?) which should be
used in case the subtest can be slow/combinatorial."
>
>> >
>> > Signed-off-by: David Weinehall <david.weinehall at linux.intel.com>
>> > ---
>> > lib/igt_core.c | 24 +++++
>> > lib/igt_core.h | 7 ++
>> > tests/gem_concurrent_blit.c | 44 ++++-----
>> > tests/kms_frontbuffer_tracking.c | 208 ++++++++++++++++++++++-----------------
>> > 4 files changed, 165 insertions(+), 118 deletions(-)
>> >
>> > diff --git a/lib/igt_core.c b/lib/igt_core.c
>> > index 59127cafe606..6575b9d6bf0d 100644
>> > --- a/lib/igt_core.c
>> > +++ b/lib/igt_core.c
>> > @@ -216,6 +216,7 @@ const char *igt_interactive_debug;
>> >
>> > /* subtests helpers */
>> > static bool list_subtests = false;
>> > +static bool with_slow_combinatorial = false;
>>
>> The option is called --all, the new subtest macro is _slow and the
>> variables and enums are called with_slow_combinatorial. Is this
>> intentional?
>
> The option is called --all because "--with-slow-combinatorial" was
> considered to be too much of a mouthful. The variables & enums are
> still retaining these names because they're much more descriptive.
Ok, let's keep is like this then (in case they don't change with
_flags suggestion).
>
> The macro is called _slow because I wanted to keep it a bit shorter,
> but I can rename it to _slow_combinatorial if that's preferred.
I agree _slow_combinatorial is too big, so let's keep it like this.
Maybe the new version is going to be _flags or something?
>
>>
>> > static char *run_single_subtest = NULL;
>> > static bool run_single_subtest_found = false;
>> > static const char *in_subtest = NULL;
>> > @@ -235,6 +236,7 @@ bool test_child;
>> >
>> > enum {
>> > OPT_LIST_SUBTESTS,
>> > + OPT_WITH_SLOW_COMBINATORIAL,
>> > OPT_RUN_SUBTEST,
>> > OPT_DESCRIPTION,
>> > OPT_DEBUG,
>> > @@ -478,6 +480,7 @@ static void print_usage(const char *help_str, bool output_on_stderr)
>> >
>> > fprintf(f, "Usage: %s [OPTIONS]\n", command_str);
>> > fprintf(f, " --list-subtests\n"
>> > + " --all\n"
>> > " --run-subtest <pattern>\n"
>> > " --debug[=log-domain]\n"
>> > " --interactive-debug[=domain]\n"
>> > @@ -510,6 +513,7 @@ static int common_init(int *argc, char **argv,
>> > int c, option_index = 0, i, x;
>> > static struct option long_options[] = {
>> > {"list-subtests", 0, 0, OPT_LIST_SUBTESTS},
>> > + {"all", 0, 0, OPT_WITH_SLOW_COMBINATORIAL},
>> > {"run-subtest", 1, 0, OPT_RUN_SUBTEST},
>> > {"help-description", 0, 0, OPT_DESCRIPTION},
>> > {"debug", optional_argument, 0, OPT_DEBUG},
>> > @@ -617,6 +621,10 @@ static int common_init(int *argc, char **argv,
>> > if (!run_single_subtest)
>> > list_subtests = true;
>> > break;
>> > + case OPT_WITH_SLOW_COMBINATORIAL:
>> > + if (!run_single_subtest)
>> > + with_slow_combinatorial = true;
>> > + break;
>> > case OPT_RUN_SUBTEST:
>> > if (!list_subtests)
>> > run_single_subtest = strdup(optarg);
>> > @@ -1629,6 +1637,22 @@ void igt_skip_on_simulation(void)
>> > igt_require(!igt_run_in_simulation());
>> > }
>> >
>> > +/**
>> > + * __igt_slow_combinatorial:
>> > + *
>> > + * This is used to skip subtests that should only be included
>> > + * when the "--all" command line option has been specified. This version
>> > + * is intended as a test.
>> > + *
>> > + * @slow_test: true if the subtest is part of the slow/combinatorial set
>> > + *
>> > + * Returns: true if the test should be run, false if the test should be skipped
>> > + */
>> > +bool __igt_slow_combinatorial(bool slow_test)
>> > +{
>> > + return !slow_test || with_slow_combinatorial;
>> > +}
>> > +
>> > /* structured logging */
>> >
>> > /**
>> > diff --git a/lib/igt_core.h b/lib/igt_core.h
>> > index 5ae09653fd55..7b592278bf6c 100644
>> > --- a/lib/igt_core.h
>> > +++ b/lib/igt_core.h
>> > @@ -191,6 +191,12 @@ bool __igt_run_subtest(const char *subtest_name);
>> > #define igt_subtest_f(f...) \
>> > __igt_subtest_f(igt_tokencat(__tmpchar, __LINE__), f)
>> >
>> > +bool __igt_slow_combinatorial(bool slow_test);
>> > +
>>
>> We also need a igt_subtest_slow() version (without "_f") and some
>> comments explaining what's the real difference between them and the
>> other macros, like the other igt_subtest_* macros.
>
> OK, fair enough.
>
>> > +#define igt_subtest_slow_f(__slow, f...) \
>> > + if (__igt_slow_combinatorial(__slow)) \
>> > + __igt_subtest_f(igt_tokencat(__tmpchar, __LINE__), f)
>>
>> Missing tab in the line above.
>
> Indeed, thanks for spotting. Will fix.
>
>> > +
>> > const char *igt_subtest_name(void);
>> > bool igt_only_list_subtests(void);
>> >
>> > @@ -669,6 +675,7 @@ void igt_disable_exit_handler(void);
>> >
>> > /* helpers to automatically reduce test runtime in simulation */
>> > bool igt_run_in_simulation(void);
>> > +
>>
>> Bad chunk.
>
> Doh, that's a remnant from moving things around. Will fix.
>
> [snip]
>
>> > diff --git a/tests/kms_frontbuffer_tracking.c b/tests/kms_frontbuffer_tracking.c
>> > index 15707b9b9040..86fd7ca08692 100644
>> > --- a/tests/kms_frontbuffer_tracking.c
>> > +++ b/tests/kms_frontbuffer_tracking.c
>> > @@ -47,8 +47,7 @@ IGT_TEST_DESCRIPTION("Test the Kernel's frontbuffer tracking mechanism and "
>> > * combinations that are somewhat redundant and don't add much value to the
>> > * test. For example, since we already do the offscreen testing with a single
>> > * pipe enabled, there's no much value in doing it again with dual pipes. If you
>> > - * still want to try these redundant tests, you need to use the --show-hidden
>> > - * option.
>> > + * still want to try these redundant tests, you need to use the --all option.
>> > *
>> > * The most important hidden thing is the FEATURE_NONE set of tests. Whenever
>> > * you get a failure on any test, it is important to check whether the same test
>> > @@ -116,6 +115,10 @@ struct test_mode {
>> > } format;
>> >
>> > enum igt_draw_method method;
>> > +
>> > + /* The test is slow and/or combinatorial;
>> > + * skip unless otherwise specified */
>> > + bool slow;
>>
>> My problem with this is that exactly none of the tests marked as
>> "slow" are actually slow here... They're either redudant or for debug
>> purposes, not slow.
>
> If they're redundant they should be removed (but that should be done by
> you or someone else who know that they are indeed redundant), as I
> already mentioned. They definitely are "slow" though, in the sense that
> running with them is slower than not running with them (admittedly the
> difference isn't comparable to that of gem_concurrent_blit, where a full
> run on my test machine took 30x as long...).
>
> If you'd like to categorise them into more categories than just
> slow/non-slow (or slow_combinatorial/non-slow_combinatorial), then by
> all means, I'll go for Thomas Wood's proposal to use the _flags
> approach instead, but for that you need to provide a patch that actually
> categorises them.
I actually like the _flags idea. It's easy to expand in case we want
to create more categories. I'm not sure if this is actually going to
be done, but I'd support it.
Bonus points if --all could become
--flags=slow,combinatorial,useless,stress,blacklisted,debug,todo,etc.
But let's not block your current patches on these things, we can leave
this for later if you want.
>
>
> Regards, David
--
Paulo Zanoni
More information about the Intel-gfx
mailing list