[igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval

[Ram Moon, AnandX]

Hi Chris,

-----Original Message-----
From: Ram Moon, AnandX 
Sent: Tuesday, November 3, 2020 11:28 PM
To: Chris Wilson <chris at chris-wilson.co.uk>; igt-dev at lists.freedesktop.org
Subject: RE: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval

Hi Chris,

Thanks for your review comment.

-----Original Message-----
From: Chris Wilson <chris at chris-wilson.co.uk>
Sent: Tuesday, November 3, 2020 4:19 PM
To: Ram Moon, AnandX <anandx.ram.moon at intel.com>; igt-dev at lists.freedesktop.org
Cc: Ram Moon, AnandX <anandx.ram.moon at intel.com>
Subject: Re: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval

Quoting Anand Moon (2020-11-02 09:08:36)
> Using confidence interval formula accuracy is estimates approximately 
> at 95% when {X} is the average of a sample of size n.
> Use this confidence interval formula as used in statistics to estimate 
> the accuracy.
> 
> Signed-off-by: Anand Moon <anandx.ram.moon at intel.com>
> ---
> See below link.
> https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule#Cumulati
> ve_distribution_function
> ---
>  tests/kms_setmode.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/kms_setmode.c b/tests/kms_setmode.c index
> 92f3ead2..58814a13 100644
> --- a/tests/kms_setmode.c
> +++ b/tests/kms_setmode.c
> @@ -494,7 +494,7 @@ static void check_timings(int crtc_idx, const drmModeModeInfo *kmode)
>          * which depend on timing, giving the lowest acceptable MTBF of 5.6s
>          * for 60Hz sampling rate.
>          */
> -       accuracy = 3. * stddev;
> +       accuracy = 3. * stddev / sqrt(CALIBRATE_TS_STEPS);

Oops, yes, this should be standard error rather than deviation. Could you add igt_stats_get_std_error() so we don't hardcode N?
-Chris

Do you want me to rename accuracy to std_error and use stats.n_values instead of CALIBRATE_TS_STEPS I could not find the definition of igt_stats_get_std_error function. 

-Anand

Thanks for fixing the code changes.

Actually above changes fix the issue to improve the accuracy calculations.
But now I am able to reproduce the original bug using 4K monitors on 5.8.x kernel, See below logs.

    CRTC[98] [Pipe A] Mode: 3840x2160 at 60Hz Connectors: DP-2[330] Expected frametime: 16668us; measured 16672.3us +- 1.805us accuracy 0.00% [0.07 scanlines]
(kms_setmode:14315) CRITICAL: Test assertion failure function check_timings, file ../tests/kms_setmode.c:544:
(kms_setmode:14315) CRITICAL: Failed assertion: fabs(mean - expected) < 1.718 * stddev
(kms_setmode:14315) CRITICAL: vblank interval differs from modeline! expected 16667.6us, measured 16672us +- 1.805us, difference 4.7us (2.6 sigma) Stack trace:
  #0 ../lib/igt_core.c:1734 __igt_fail_assert()
  #1 ../tests/kms_setmode.c:544 test_one_combination()
  #2 ../tests/kms_setmode.c:834 __real_main883()
  #3 ../tests/kms_setmode.c:883 main()
  #4 ../csu/libc-start.c:342 __libc_start_main()
  #5 [_start+0x2e]
Subtest basic failed.
**** DEBUG ****
 
So after in digging into the below comments it turns out that *confidence interval* with respect to standard deviation is calculated with *Cumulative distribution function* (formula)

As per the Wikipedia table
For various values of z, the percentage of values expected to lie in and outside the symmetric interval, CI = (−zσ, zσ), are as follows:
[0] https://en.wikipedia.org/wiki/Standard_deviation#Rules_for_normally_distributed_data

It turns out that  *3 σ* gives around proportion percentage 99.7300204% which is according to the CI = (−zσ, zσ) table.

I would like to submit a below patch as this changes fix the overall bug.
Please share your thought and suggestion on any alternative approach I should try.

Thanks
Anand

naveen at naveen-GB-Z390-Garuda:~/anand/igt-gpu-tools.internal-rebasing$ git diff diff --git a/tests/kms_setmode.c b/tests/kms_setmode.c index 928a5a8cf..58da36985 100644
--- a/tests/kms_setmode.c
+++ b/tests/kms_setmode.c
@@ -526,7 +526,7 @@ static void check_timings(int crtc_idx, const drmModeModeInfo *kmode)
         * See:
         * https://en.wikipedia.org/wiki/Standard_deviation#Rules_for_normally_distributed_data
         */
-       igt_assert_f(fabs(mean - expected) < 1.718 * stddev,
+       igt_assert_f(fabs(mean - expected) < 3. * stddev,
                     "vblank interval differs from modeline! expected %.1fus, measured %1.fus +- %.3fus, difference %.1fus (%.1f sigma)\n",
                     expected, mean, stddev,
                     fabs(mean - expected), fabs(mean - expected) / stddev);