[Mesa-dev] [PATCH shader-db 4/4] si-report.py: add completely new shader statistics reporting

Marek Olšák maraeo at gmail.com
Fri Jul 22 20:39:00 UTC 2016


For those who would like to try this out, FYI, I've pushed this series just now.

Marek

On Thu, Jul 14, 2016 at 4:28 PM, Marek Olšák <maraeo at gmail.com> wrote:
> From: Marek Olšák <marek.olsak at amd.com>
>
> We can remove the old reporting if people are OK with that.
> The old reporting has a bug that it reports 0->N changes as 0 % in several
> places. (should be inf %)
>
> The new reporting shows:
> - VGPR spilling shaders and apps (from the second file only)
> - the same for SGPRs
> - worst regressions (from the comparison of both files)
> - percentage deltas at the end
> .. and colors!!!
>
> Example:
>
>  WORST VGPR SPILLS (not deltas)                                        VGPRs SpillVGPR ScratchVGPR
>  shaders/private/f1-2015/18.shader_test [0]                               32         0       516
>  shaders/private/bioshock-infinite/256.shader_test [0]                    64       176       180
>  shaders/private/ue4_lightroom_interior_day/42.shader_test [0]            28         0        76
>  shaders/private/dirt-showdown/676.shader_test [0]                        68        49        72
>  shaders/private/f1-2015/1102.shader_test [0]                             52         0        72
>  shaders/private/bioshock-infinite/814.shader_test [0]                    64        57        60
>  shaders/private/ue4_lightroom_interior_day/33.shader_test [0]            24         0        52
>  shaders/private/ue4_lightroom_interior_day/37.shader_test [0]            24         0        52
>  shaders/private/bioshock-infinite/698.shader_test [0]                    16         0        36
>  shaders/private/dirt-showdown/406.shader_test [0]                        64        33        36
>
>  VGPR SPILLING APPS   Shaders SpillVGPR ScratchVGPR
>  alien_isolation         2938        12        16
>  bioshock-infinite       1769       233       720
>  dirt-showdown            541        82       108
>  f1-2015                  774         0       624
>  tesseract                430         2         4
>  ue4_lightroom_inter..     74         0       180
>
>  WORST SGPR SPILLS (not deltas)                                        SGPRs SpillSGPR
>  shaders/private/talos_principle/1942.shader_test [1]                     80       168
>  shaders/private/ue4_effects_cave/289.shader_test [0]                     80       168
>  shaders/private/talos_principle/2052.shader_test [1]                     80       161
>  shaders/private/serious_sam_3_bfe/1081.shader_test [1]                   80       148
>  shaders/private/borderlands2/5330.shader_test [0]                        80       137
>  shaders/private/talos_principle/2040.shader_test [1]                     80       133
>  shaders/private/talos_principle/2041.shader_test [1]                     80       133
>  shaders/private/talos_principle/2036.shader_test [1]                     80       133
>  shaders/private/talos_principle/2035.shader_test [1]                     80       133
>  shaders/private/borderlands2/5548.shader_test [0]                        80       131
>
>  SGPR SPILLING APPS   Shaders SpillSGPR AvgPerSh
>  alien_isolation         2938     23198      7.9
>  batman_arkham_origins    589        30      0.1
>  bioshock-infinite       1769        84      0.0
>  borderlands2            3968      6449      1.6
>  brutal-legend            338       647      1.9
>  civilization_beyond..    116       213      1.8
>  counter_strike_glob..   1142      4338      3.8
>  dirt-showdown            541      1071      2.0
>  dolphin                   22        62      2.8
>  dota2                   1747       338      0.2
>  europa_universalis_4      76        44      0.6
>  f1-2015                  774      6245      8.1
>  left_4_dead_2           1762     13778      7.8
>  metro_2033_redux        2670       547      0.2
>  nexuiz                    80       111      1.4
>  portal                   474      2211      4.7
>  serious_sam_3_bfe        392      6626     16.9
>  talos_principle          324      4539     14.0
>  team_fortress_2          808      4823      6.0
>  thea                     172        41      0.2
>  ue4_effects_cave         299       494      1.7
>  ue4_elemental            586       355      0.6
>  ue4_lightroom_inter..     74        29      0.4
>  ue4_realistic_rende..     92        60      0.7
>  unigine_heaven           322       174      0.5
>  unigine_sanctuary        264       400      1.5
>  unigine_tropics          210       328      1.6
>  unigine_valley           278       427      1.5
>  unity                     72        45      0.6
>  warsow                   176        13      0.1
>  witcher2                1040         2      0.0
>
>  WORST REGRESSIONS - VGPRS                                            Before     After     Delta Percentage
>  shaders/private/metro_2033_redux/1082.shader_test [0]                   136       180        44   32.35 %
>  shaders/private/witcher2/20.shader_test [0]                              48        64        16   33.33 %
>  shaders/private/witcher2/1097.shader_test [0]                            60        76        16   26.67 %
>  shaders/private/unigine_valley/298.shader_test [0]                       44        56        12   27.27 %
>  shaders/private/witcher2/158.shader_test [0]                             52        64        12   23.08 %
>  shaders/private/witcher2/138.shader_test [0]                             60        72        12   20.00 %
>  shaders/private/witcher2/136.shader_test [0]                             60        72        12   20.00 %
>  shaders/private/witcher2/187.shader_test [0]                             56        68        12   21.43 %
>  shaders/private/left_4_dead_2/1567.shader_test [0]                       32        40         8   25.00 %
>  shaders/private/team_fortress_2/4662.shader_test [0]                     20        28         8   40.00 %
>
>  WORST REGRESSIONS - Spilled SGPRs                                    Before     After     Delta Percentage
>  shaders/private/serious_sam_3_bfe/883.shader_test [0]                    14        49        35  250.00 %
>  shaders/private/talos_principle/1942.shader_test [0]                     14        49        35  250.00 %
>  shaders/private/serious_sam_3_bfe/1081.shader_test [0]                   14        49        35  250.00 %
>  shaders/private/talos_principle/1941.shader_test [0]                     14        49        35  250.00 %
>  shaders/private/talos_principle/2052.shader_test [0]                      9        42        33  366.67 %
>  shaders/private/serious_sam_3_bfe/869.shader_test [0]                    35        67        32   91.43 %
>  shaders/private/talos_principle/1986.shader_test [0]                     45        76        31   68.89 %
>  shaders/private/left_4_dead_2/3764.shader_test [0]                       15        46        31  206.67 %
>  shaders/private/talos_principle/2008.shader_test [0]                     45        76        31   68.89 %
>  shaders/private/talos_principle/2035.shader_test [0]                     45        76        31   68.89 %
>
>  WORST REGRESSIONS - Spilled VGPRs                                    Before     After     Delta Percentage
>  shaders/tesseract/506.shader_test [0]                                     0         2         2     inf %
>
>  WORST REGRESSIONS - Scratch VGPRs                                    Before     After     Delta Percentage
>  shaders/tesseract/506.shader_test [0]                                     0         4         4     inf %
>
>  WORST REGRESSIONS - Code Size                                        Before     After     Delta Percentage
>  shaders/private/talos_principle/1942.shader_test [0]                   3116      3808       692   22.21 %
>  shaders/private/talos_principle/1941.shader_test [0]                   3116      3808       692   22.21 %
>  shaders/private/serious_sam_3_bfe/883.shader_test [0]                  3108      3792       684   22.01 %
>  shaders/private/serious_sam_3_bfe/1081.shader_test [0]                 3108      3792       684   22.01 %
>  shaders/private/serious_sam_3_bfe/869.shader_test [0]                  4124      4744       620   15.03 %
>  shaders/private/talos_principle/2052.shader_test [0]                   2904      3512       608   20.94 %
>  shaders/private/left_4_dead_2/3764.shader_test [0]                     3344      3928       584   17.46 %
>  shaders/private/left_4_dead_2/1260.shader_test [0]                     3344      3928       584   17.46 %
>  shaders/private/talos_principle/2061.shader_test [0]                   5312      5892       580   10.92 %
>  shaders/private/talos_principle/1994.shader_test [0]                   5312      5892       580   10.92 %
>
>  PERCENTAGE DELTAS    Shaders     SGPRs     VGPRs SpillSGPR SpillVGPR   Scratch  CodeSize  MaxWaves     Waits
>  (unknown)                  4     .         .         .         .         .       -0.02 %     .         .
>  0ad                        6     .         .         .         .         .         .         .         .
>  alien_isolation         2938     .       -5.35 %   -0.77 %     .         .        0.21 %    3.79 %     .
>  anholt                    10     .         .         .         .         .         .         .         .
>  batman_arkham_origins    589     .       -4.77 %  -96.56 %     .         .       -1.59 %    4.48 %     .
>  bioshock-infinite       1769     .       -1.41 %  -86.60 %     .         .       -0.59 %    0.94 %     .
>  borderlands2            3968     .       -2.20 %  -36.23 %     .         .       -1.07 %    1.15 %     .
>  brutal-legend            338     .       -0.90 %   -6.50 %     .         .       -0.10 %    0.43 %     .
>  civilization_beyond..    116     .       -0.97 %    0.47 %     .         .        0.04 %    0.46 %     .
>  counter_strike_glob..   1142     .       -2.00 %   -7.58 %     .         .       -0.43 %    0.06 %     .
>  dirt-showdown            541     .       -1.37 %  -22.95 %     .       -3.57 %   -0.79 %    1.03 %     .
>  dolphin                   22     .         .         .         .         .        0.23 %     .         .
>  dota2                   1747     .       -0.07 %     .         .         .        0.03 %    0.04 %     .
>  europa_universalis_4      76     .       -0.46 %     .         .         .        0.09 %     .         .
>  f1-2015                  774     .       -2.11 %   -1.51 %     .         .       -0.02 %    1.05 %     .
>  furmark-0.7.0              4     .         .         .         .         .        0.16 %     .         .
>  gimark-0.7.0              10     .         .         .         .         .        0.18 %     .         .
>  glamor                    16     .         .         .         .         .       -0.91 %     .         .
>  humus-celshading           4     .         .         .         .         .        0.42 %     .         .
>  humus-domino               6     .         .         .         .         .         .         .         .
>  humus-dynamicbranching    24     .       -0.70 %     .         .         .        0.42 %    0.45 %     .
>  humus-hdr                 10     .         .         .         .         .         .         .         .
>  humus-portals              2     .         .         .         .         .         .         .         .
>  humus-volumetricfog..      6     .         .         .         .         .         .         .         .
>  left_4_dead_2           1762     .       -0.62 %   10.13 %     .         .        0.72 %   -0.14 %     .
>  metro_2033_redux        2670     .       -4.73 %  -12.20 %     .         .        0.20 %    1.52 %     .
>  nexuiz                    80     .         .         .         .         .         .         .         .
>  pixmark-julia-fp32         2     .         .         .         .         .         .         .         .
>  pixmark-julia-fp64         2     .         .         .         .         .         .         .         .
>  pixmark-piano-0.7.0        2     .         .         .         .         .       -0.85 %     .         .
>  pixmark-volplosion-..      2     .         .         .         .         .         .         .         .
>  plot3d-0.7.0               8     .         .         .         .         .         .         .         .
>  portal                   474     .       -2.05 %  206.23 %     .         .        1.69 %    0.58 %     .
>  sauerbraten                7     .         .         .         .         .        0.62 %     .         .
>  serious_sam_3_bfe        392     .       -8.64 %    0.93 %     .         .       -1.79 %    4.05 %     .
>  supertuxkart               4     .         .         .         .         .         .         .         .
>  talos_principle          324     .       -7.67 %  -10.07 %     .         .       -1.39 %    2.92 %     .
>  team_fortress_2          808     .       -2.62 %  276.21 %     .         .        2.26 %    1.24 %     .
>  tesseract                430     .       -1.31 % -100.00 %     inf %     inf %   -0.12 %    0.75 %     .
>  tessmark-0.7.0             6     .         .         .         .         .        0.09 %     .         .
>  thea                     172     .       -1.98 %     .         .         .        0.34 %    0.91 %     .
>  ue4_effects_cave         299     .       -1.87 %   -8.52 %     .         .       -0.18 %    1.87 %     .
>  ue4_elemental            586     .       -4.67 %  -42.56 %     .         .       -1.17 %    4.57 %     .
>  ue4_lightroom_inter..     74     .       -1.57 %  -12.12 %     .         .        0.12 %    1.00 %     .
>  ue4_realistic_rende..     92     .       -0.27 %     .         .         .        0.11 %     .         .
>  unigine_heaven           322     .       -0.87 %  -36.73 %     .         .       -0.46 %    0.69 %     .
>  unigine_sanctuary        264     .       -1.57 %   -7.41 %     .         .        0.01 %    0.96 %     .
>  unigine_tropics          210     .       -1.22 %  -10.87 %     .         .       -0.18 %    0.88 %     .
>  unigine_valley           278     .       -2.11 %  -27.87 %     .         .       -0.46 %    1.39 %     .
>  unity                     72     .       -1.74 %  -15.09 %     .         .       -0.54 %    0.93 %     .
>  warsow                   176     .       -0.10 %     .         .         .         .        0.06 %     .
>  warzone2100                4     .       -4.35 %     .         .         .        0.67 %    2.63 %     .
>  witcher2                1040     .       -5.93 %  -93.55 %     .         .       -0.59 %    2.54 %     .
>  xcom_enemy_within       1236     .       -4.81 % -100.00 %     .         .       -0.03 %    3.22 %     .
>  yofrankie                 82     .       -0.41 %     .         .         .        0.80 %    0.14 %     .
>  ------------------------------------------------------------------------------------------------------------
>  All affected            7287     .       -8.94 %   -5.18 %    0.61 %     .       -0.36 %    6.21 %     .
>  ------------------------------------------------------------------------------------------------------------
>  Total                  26002     .       -3.09 %   -1.82 %    0.61 %     .       -0.16 %    1.55 %     .
> ---
>  si-report.py | 253 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 252 insertions(+), 1 deletion(-)
>
> diff --git a/si-report.py b/si-report.py
> index 523c452..cc3d4ce 100755
> --- a/si-report.py
> +++ b/si-report.py
> @@ -28,6 +28,11 @@ import itertools
>  import re
>  import sys
>
> +set_red = "\033[31m"
> +set_green = "\033[1;32m"
> +set_yellow = "\033[1;33m"
> +set_normal = "\033[0m"
> +
>  def format_float(f, suffix = ' %'):
>      return "{0:0.2f}{1}".format(f, suffix)
>
> @@ -42,6 +47,23 @@ def calculate_percent_change(b, a):
>          return 0 if a == 0 else float("inf")
>      return 100 * float(a - b) / float(b)
>
> +def format_table_cell(n, more_is_better = False, colored = True, is_percent = False):
> +    if is_percent and abs(n) < 0.01:
> +        return "     .    "
> +
> +    str =  ("{:>8.2f} %" if is_percent else "{:>10}").format(n)
> +    if colored:
> +        if n > 0.5:
> +            str = (set_green if more_is_better else set_red) + str + set_normal
> +        elif n < -0.5:
> +            str = (set_red if more_is_better else set_green) + str + set_normal
> +    return str
> +
> +
> +def format_percent_change(b, a, more_is_better = False, colored = True):
> +    percent = calculate_percent_change(b, a)
> +    return format_table_cell(percent, more_is_better, colored, is_percent = True)
> +
>  def cmp_max_unit(current, comp):
>      return comp[0] > current[0]
>
> @@ -252,6 +274,22 @@ def compare_stats(before, after):
>          result.__dict__[name] = (a - b, b, a)
>      return result
>
> +def subtract_stats(x, y):
> +    result = si_stats()
> +    for name in result.get_metrics():
> +        result.__dict__[name] = x.__dict__[name] - y.__dict__[name]
> +    return result
> +
> +def is_regression(before, after):
> +    for field in before.get_metrics():
> +        if field == 'maxwaves':
> +            if before.__dict__[field] > after.__dict__[field]:
> +                return True
> +        else:
> +            if before.__dict__[field] < after.__dict__[field]:
> +                return True
> +    return False
> +
>  def divide_stats(num, div):
>      result = si_stats()
>      for name in result.get_metrics():
> @@ -411,11 +449,224 @@ def compare_results(before_all_results, after_all_results):
>          print "*** Compile errors encountered! (before: {}, after: {})".format(
>              num_before_errors, num_after_errors)
>
> +class grouped_stats:
> +    def __init__(self):
> +        self.num_shaders = 0
> +        self.before = si_stats()
> +        self.after = si_stats()
> +        self.diff = si_stats()
> +
> +    def add(self, before, after):
> +        self.num_shaders += 1
> +        self.before.add(before)
> +        self.after.add(after)
> +
> +    def set_one_shader(self, before, after):
> +        self.before = before
> +        self.after = after
> +        self.diff = subtract_stats(after, before)
> +
> +    def print_vgpr_spilling_app(self, name):
> +        if (self.after.spilled_vgprs > 0 or
> +            self.after.scratch_vgprs > 0):
> +            print " {:22}{:6}{:10}{:10}".format(
> +                name,
> +                self.num_shaders,
> +                self.after.spilled_vgprs,
> +                self.after.scratch_vgprs)
> +
> +    def print_one_shader_vgpr_spill(self, name):
> +        if (self.after.spilled_vgprs > 0 or
> +            self.after.scratch_vgprs > 0):
> +            print " {:65}{:10}{:10}{:10}".format(
> +                name,
> +                self.after.vgprs,
> +                self.after.spilled_vgprs,
> +                self.after.scratch_vgprs)
> +
> +    def print_sgpr_spilling_app(self, name):
> +        if self.after.spilled_sgprs > 0:
> +            print " {:22}{:6}{:10}{:>9.1f}".format(
> +                name,
> +                self.num_shaders,
> +                self.after.spilled_sgprs,
> +                float(self.after.spilled_sgprs) / float(self.num_shaders))
> +
> +    def print_one_shader_sgpr_spill(self, name):
> +        if self.after.spilled_sgprs > 0:
> +            print " {:65}{:10}{:10}".format(
> +                name,
> +                self.after.sgprs,
> +                self.after.spilled_sgprs)
> +
> +    def print_percentages(self, name):
> +        print " {:22}{:6}{}{}{}{}{}{}{}{}".format(
> +            name,
> +            self.num_shaders,
> +            format_percent_change(self.before.sgprs, self.after.sgprs),
> +            format_percent_change(self.before.vgprs, self.after.vgprs),
> +            format_percent_change(self.before.spilled_sgprs, self.after.spilled_sgprs),
> +            format_percent_change(self.before.spilled_vgprs, self.after.spilled_vgprs),
> +            format_percent_change(self.before.scratch_vgprs, self.after.scratch_vgprs),
> +            format_percent_change(self.before.code_size, self.after.code_size),
> +            format_percent_change(self.before.maxwaves, self.after.maxwaves, more_is_better = True),
> +            format_percent_change(self.before.waitstates, self.after.waitstates))
> +
> +    def print_regression(self, name, field):
> +        print " {:65}{:10}{:10}{}{}".format(
> +            name,
> +            self.before.__dict__[field],
> +            self.after.__dict__[field],
> +            format_table_cell(self.after.__dict__[field] - self.before.__dict__[field]),
> +            format_percent_change(self.before.__dict__[field], self.after.__dict__[field]))
> +
> +"""
> +Return "filename [index]", because files can contain multiple shaders.
> +"""
> +def get_shader_name(list, orig):
> +    for i in range(10):
> +        # add the index to the name
> +        name = orig + " [{}]".format(i)
> +        if name not in list:
> +                return name
> +    assert False
> +    return "(error)"
> +
> +
> +def print_yellow(str):
> +    print set_yellow + str + set_normal
> +
> +def print_tables(before_all_results, after_all_results):
> +    re_app = re.compile(r"^.*/([^/]+)/[^/]+$")
> +
> +    apps = defaultdict(grouped_stats)
> +    shaders = defaultdict(grouped_stats)
> +    total = grouped_stats()
> +    total_affected = grouped_stats()
> +
> +    all_files = set(itertools.chain(before_all_results.keys(),
> +                                    after_all_results.keys()))
> +
> +    for file in all_files:
> +        # get the application name (inner-most directory)
> +        match_app = re_app.match(file)
> +        if match_app is None:
> +            app = "(unknown)"
> +        else:
> +            app = match_app.group(1)
> +        if len(app) > 22:
> +            app = app[0:19] + ".."
> +
> +        before_test_results = before_all_results.get(file)
> +        after_test_results = after_all_results.get(file)
> +
> +        if before_test_results is None or after_test_results is None:
> +            continue
> +
> +        for before, after in zip(before_test_results, after_test_results):
> +            if after.error or before.error:
> +                continue
> +
> +            apps[app].add(before, after)
> +            total.add(before, after)
> +
> +            if not subtract_stats(before, after).is_empty():
> +                total_affected.add(before, after)
> +
> +            # we don't have to add all shaders, just those that we may need
> +            # to display
> +            if (is_regression(before, after) or
> +                after.scratch_vgprs > 0 or
> +                after.spilled_vgprs > 0 or
> +                after.spilled_sgprs > 0):
> +                name = get_shader_name(shaders, file)
> +                shaders[name].set_one_shader(before, after)
> +
> +    # worst VGPR spills
> +    num = 0
> +    sort_key = lambda v: -v[1].after.scratch_vgprs
> +    for name, stats in sorted(shaders.items(), key = sort_key):
> +        if num == 0:
> +            print_yellow(" WORST VGPR SPILLS (not deltas)" + (" " * 40) +
> +                         "VGPRs SpillVGPR ScratchVGPR")
> +        stats.print_one_shader_vgpr_spill(name)
> +        num += 1
> +        if num == 10:
> +            break
> +    if num > 0:
> +        print
> +
> +    # VGPR spilling apps
> +    print_yellow(" VGPR SPILLING APPS   Shaders SpillVGPR ScratchVGPR")
> +    for name, stats in sorted(apps.items()):
> +        stats.print_vgpr_spilling_app(name)
> +    print
> +
> +    # worst SGPR spills
> +    num = 0
> +    sort_key = lambda v: -v[1].after.spilled_sgprs
> +    for name, stats in sorted(shaders.items(), key = sort_key):
> +        if num == 0:
> +            print_yellow(" WORST SGPR SPILLS (not deltas)" + (" " * 40) +
> +                         "SGPRs SpillSGPR")
> +        stats.print_one_shader_sgpr_spill(name)
> +        num += 1
> +        if num == 10:
> +            break
> +    if num > 0:
> +        print
> +
> +    # SGPR spilling apps
> +    print_yellow(" SGPR SPILLING APPS   Shaders SpillSGPR AvgPerSh")
> +    for name, stats in sorted(apps.items()):
> +        stats.print_sgpr_spilling_app(name)
> +    print
> +
> +    # worst regressions
> +    metrics = si_stats().metrics
> +    for i in range(len(metrics)):
> +        # maxwaves regressions aren't reported (see vgprs/sgprs instead)
> +        if metrics[i][0] == 'maxwaves':
> +            continue
> +
> +        field = metrics[i][0]
> +        num = 0
> +        sort_key = lambda v: -v[1].diff.__dict__[field]
> +
> +        for name, stats in sorted(shaders.items(), key = sort_key):
> +            if stats.diff.__dict__[field] <= 0:
> +                continue
> +
> +            if num == 0:
> +                print_yellow(" WORST REGRESSIONS - {:49}".format(metrics[i][1]) +
> +                             "Before     After     Delta Percentage")
> +            stats.print_regression(name, field)
> +            num += 1
> +            if num == 10:
> +                break
> +        if num > 0:
> +            print
> +
> +    # percentages
> +    legend = "Shaders     SGPRs     VGPRs SpillSGPR SpillVGPR   Scratch  CodeSize  MaxWaves     Waits"
> +    print_yellow(" PERCENTAGE DELTAS    " + legend)
> +    for name, stats in sorted(apps.items()):
> +        stats.print_percentages(name)
> +    print " " + ("-" * (21 + len(legend)))
> +    total_affected.print_percentages("All affected")
> +    print " " + ("-" * (21 + len(legend)))
> +    total.print_percentages("Total")
> +    print
> +
>  def main():
>      before = sys.argv[1]
>      after = sys.argv[2]
>
> -    compare_results(get_results(before), get_results(after))
> +    results_before = get_results(before)
> +    results_after = get_results(after)
> +
> +    compare_results(results_before, results_after)
> +    print_tables(results_before, results_after)
>
>  if __name__ == "__main__":
>      main()
> --
> 2.7.4
>


More information about the mesa-dev mailing list