[igt-dev] [PATCH i-g-t] tests/kms_setmode: Fix dynamic subtests

[B, Jeevan]

> -----Original Message-----
> From: Modem, Bhanuprakash <bhanuprakash.modem at intel.com>
> Sent: Friday, June 23, 2023 9:20 AM
> To: B, Jeevan <jeevan.b at intel.com>; igt-dev at lists.freedesktop.org
> Cc: kamil.konieczny at linux.intel.com
> Subject: Re: [PATCH i-g-t] tests/kms_setmode: Fix dynamic subtests
> 
> Hi Jeevan,
> 
> On Thu-22-06-2023 04:46 pm, Jeevan B wrote:
> > basic-clone-single-crtc and invalid-clone-exclusive-crtc were not part
> > of dynamic subtest so tests were getting skipped. fixed the test to
> > execute skipping tests as expected.
> 
> What does it mean these tests are not part of dynamic subtests?
> 
> AFAIK, all basic-* tests are positive test cases & there are corresponding
> negative test cases (invalid-*)
> 
> If any positive test executed, the corresponding negative test is supposed to
> skip & vice versa.


Hi Bhanu,

Got it. Tests before converting to dynamic. 

Starting subtest: basic-clone-single-crtc
Testing: basic-clone-single-crtc 2 connector combinations
Subtest basic-clone-single-crtc: SUCCESS (0.242s)

Starting subtest: invalid-clone-single-crtc
Testing: invalid-clone-single-crtc 2 connector combinations
  Test id#1 CRTC count 1
    CRTC[80] [Pipe A] Mode: 1920x1080 at 60Hz Connectors: DP-1[236], HDMI-A-2[255]
  Test id#2 CRTC count 1
    CRTC[131] [Pipe B] Mode: 1920x1080 at 60Hz Connectors: DP-1[236], HDMI-A-2[255]
Subtest invalid-clone-single-crtc: SUCCESS (0.371s)

Starting subtest: invalid-clone-exclusive-crtc
Testing: invalid-clone-exclusive-crtc 2 connector combinations
Subtest invalid-clone-exclusive-crtc: SUCCESS (0.330s)

Starting subtest: clone-exclusive-crtc
Testing: clone-exclusive-crtc 2 connector combinations
  Test id#1 CRTC count 2
    CRTC[80] [Pipe A] Mode: 3840x2160 at 60Hz Connectors: DP-1[236]
    CRTC[131] [Pipe B] Mode: 1920x1200 at 60Hz Connectors: HDMI-A-2[255]
  Test id#2 CRTC count 2
    CRTC[131] [Pipe B] Mode: 3840x2160 at 60Hz Connectors: DP-1[236]
    CRTC[80] [Pipe A] Mode: 1920x1200 at 60Hz Connectors: HDMI-A-2[255]
Subtest clone-exclusive-crtc: SUCCESS (1.282s)

So, when tests now started to skip these are getting caught by filters. 
Should these skips be kept as expected behavior or should we modify tests ?? 

Thanks 
Jeevan B  
> 
> Example:
> https://intel-gfx-ci.01.org/tree/drm-tip/shards-
> all.html?testfilter=kms_setmode&hosts=shard-glk
> 
> 
> - basic-close-single-crtc: SKIP
> - invalid-clone-single-crtc: PASS
> 
> - clone-executive-crtc: PASS
> - invalid-clone-executive-crtc: SKIP
> 
> >
> > v2: add missing call.
> >
> > Signed-off-by: Jeevan B <jeevan.b at intel.com>
> > ---
> >   tests/kms_setmode.c | 26 +++++++++++++-------------
> >   1 file changed, 13 insertions(+), 13 deletions(-)
> >
> > diff --git a/tests/kms_setmode.c b/tests/kms_setmode.c index
> > a611d938..96fb0b4b 100644
> > --- a/tests/kms_setmode.c
> > +++ b/tests/kms_setmode.c
> > @@ -647,20 +647,19 @@ static void test_one_combination(const struct
> test_config *tconf,
> >   	struct crtc_config crtcs[MAX_CRTCS];
> >   	int crtc_count;
> >   	bool config_valid;
> > +	int i, pos = 0;
> > +	char test_name[256];
> >
> >   	setup_crtcs(tconf, cconfs, connector_count, crtcs,
> >   		    &crtc_count, &config_valid);
> >
> > -	if (config_valid == !(tconf->flags & TEST_INVALID)) {
> > -		int i, pos = 0;
> > -		char test_name[256];
> > -
> > -		for (i = 0; i < crtc_count; i++) {
> > -			if (i > 0)
> > -				pos += snprintf(&test_name[pos],
> ARRAY_SIZE(test_name) - pos, "-");
> > -			pos += get_test_name_str(&crtcs[i],
> &test_name[pos], ARRAY_SIZE(test_name) - pos);
> > -		}
> > +	for (i = 0; i < crtc_count; i++) {
> > +		if (i > 0)
> > +			pos += snprintf(&test_name[pos],
> ARRAY_SIZE(test_name) - pos, "-");
> > +		pos += get_test_name_str(&crtcs[i], &test_name[pos],
> ARRAY_SIZE(test_name) - pos);
> > +	}
> >
> > +	if (config_valid == !(tconf->flags & TEST_INVALID)) {
> >   		for (i = 0; i < crtc_count; i++) {
> >   			struct crtc_config *crtc = &crtcs[i];
> >
> > @@ -679,16 +678,17 @@ static void test_one_combination(const struct
> test_config *tconf,
> >   			    ((i > 0) && (crtc[i - 1].mode.hdisplay >
> MAX_HDISPLAY_PER_CRTC) &&
> >   			     (abs(crtc->crtc_idx - crtcs[i - 1].crtc_idx) <= 1))) {
> >   				igt_info("Combo: %s is not possible with
> selected mode(s).\n", test_name);
> > -				goto out;
> > +				cleanup_crtcs(crtcs, crtc_count);
> > +				return;
> >   			}
> >   		}
> >
> >   		igt_dynamic_f("%s", test_name)
> >   			test_crtc_config(tconf, crtcs, crtc_count);
> > +	} else {
> > +		igt_dynamic_f("%s", test_name)
> > +			cleanup_crtcs(crtcs, crtc_count);
> 
> We are supposed to execute the test, not a cleanup. This will always throw
> the test result as PASS (which is bogus) without even testing anything.
> 
> - Bhanu
> 
> >   	}
> > -
> > -out:
> > -	cleanup_crtcs(crtcs, crtc_count);
> >   }
> >
> >   static int assign_crtc_to_connectors(const struct test_config
> > *tconf,