[igt-dev] [PATCH i-g-t 0/6] Dynamic subtests

[Daniel Vetter]

On Mon, Jul 22, 2019 at 03:32:33PM +0300, Petri Latvala wrote:
> On Fri, Jun 21, 2019 at 10:26:31AM +0100, Tvrtko Ursulin wrote:
> > I am confused by all this. Could we then achieve the same result by just
> > adding the ability to hide subtests from test enumeration?
> > 
> > igt_subtest_group {
> > 	igt_subtest("a");
> > }
> > 
> > igt_dynamic_subtest_group("group") {
> > 	igt_subtest("b");
> > }
> > 
> > # ./test --list-subtests
> > a
> > group
> > 
> > # .test --list-all-subtests
> > a
> > group-b
> > 
> > # ./test --r group
> > PASS
> > 
> > # ./test --r group-b
> > PASS
> > 
> > Would the above satisfy CI requirements? Sounds like it would be easier to
> > implement wrt needed code base changes in individual tests and possibly the
> > IGT core code would not be difficult either? Just needs to track in what
> > kind of subgroup it is to decide what to do when listing/running tests.
> 
> 
> With sufficient fiddling of names, this is exactly what this series does.
> 
> 
> igt_subtest_group {
> 	igt_subtest("a");
> }
> 
> igt_dynamic_subtest_container("group") {
> 	igt_dynamic_subtest("b");
> }
> 
> # ./test --list-subtests
> a
> group
> 
> # .test --list-all-subtests
> error: nope
> 
> # ./test --r group
> Dynamic "b": PASS
> PASS
> 
> # ./test --r group --dynamic-subtest b
> Dynamic "b": PASS
> PASS
> 
> 
> 
> The listing of dynamic subtests is not implemented as that is
> definitely not trivial.

I think trying to list dynamic subtests without running them defeats the
entire point here. At least it would get really tricky to tell "hw/driver
acccess necessary to enumerate tests" from "hw/driver access that actually
tests something". E.g. with atomic TEST_ONLY you might have to actually
construct all the possible configs to figure out which ones are supported.
-Daniel

> 
> 
> -- 
> Petri Latvala
> _______________________________________________
> igt-dev mailing list
> igt-dev at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/igt-dev

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch