[Mesa-dev] [PATCH 42/47] i965/fs: Optimize 16-bit SSBO stores by packing two into a 32-bit reg

[Alejandro Piñeiro]

From: Eduardo Lima Mitev <elima at igalia.com>

Currently, we use byte-scattered write messages for storing 16-bit
into an SSBO. This is because untyped surface messages have a fixed
32-bit size.

This patch optimizes these 16-bit writes by combining 2 values (e.g,
two consecutive components) into a 32-bit register, packing the two
16-bit words.

16-bit single component values will continue to use byte-scattered
write messages.

This optimization reduces the number of SEND messages used for storing
16-bit values potentially by half, which cuts down execution time
significantly because byte-scattered writes are an expensive
operation.
---
 src/intel/compiler/brw_fs_nir.cpp | 65 +++++++++++++++++++++++++++++++++------
 1 file changed, 56 insertions(+), 9 deletions(-)

diff --git a/src/intel/compiler/brw_fs_nir.cpp b/src/intel/compiler/brw_fs_nir.cpp
index 76f1359b164..63191ef3746 100644
--- a/src/intel/compiler/brw_fs_nir.cpp
+++ b/src/intel/compiler/brw_fs_nir.cpp
@@ -4222,6 +4222,9 @@ fs_visitor::nir_emit_intrinsic(const fs_builder &bld, nir_intrinsic_instr *instr
          unsigned first_component = ffs(writemask) - 1;
          unsigned length = ffs(~(writemask >> first_component)) - 1;
 
+         fs_reg current_val_reg =
+            offset(val_reg, bld, first_component * type_slots);
+
          /* We can't write more than 2 64-bit components at once. Limit the
           * length of the write to what we can do and let the next iteration
           * handle the rest
@@ -4229,11 +4232,45 @@ fs_visitor::nir_emit_intrinsic(const fs_builder &bld, nir_intrinsic_instr *instr
          if (type_size > 4) {
             length = MIN2(2, length);
          } else if (type_size == 2) {
-            /* For 16-bit types we are using byte scattered writes, that can
-             * only write one component per call. So we limit the length, and
-             * let the write happening in several iterations.
+            /* For 16-bit types we pack two consecutive values into a
+             * 32-bit word and use an untyped write message. For single values
+             * we need to use byte-scattered writes because untyped writes work
+             * on multiples of 32 bits.
+             *
+             * For example, if there is a 3-component vector we submit one
+             * untyped-write message of 32-bit (first two components), and one
+             * byte-scattered write message (the last component).
              */
-            length = 1;
+            if (length >= 2) {
+               /* pack two consecutive 16-bit words into a 32-bit register,
+                * using the same original source register.
+                */
+               fs_reg dest = retype(current_val_reg, BRW_REGISTER_TYPE_UW);
+               dest.stride = 0;
+
+               fs_reg src = retype(val_reg, BRW_REGISTER_TYPE_UW);
+               src.stride = 2;
+
+               unsigned writemask_tmp = writemask;
+               for (unsigned i = 0; i < 2; i++) {
+                  unsigned first_component_tmp = ffs(writemask_tmp) - 1;
+
+                  fs_reg tmp = offset(dest, bld, i);
+                  tmp.stride = 2;
+
+                  bld.MOV(tmp,
+                          offset(src, bld, first_component_tmp * type_slots));
+
+                  writemask_tmp &= (15 << (first_component_tmp + 1));
+               }
+
+               length = 2;
+            } else {
+               /* For single 16-bit values, we just limit the length to 1 and
+                * use a byte-scattered write message below.
+                */
+               length = 1;
+            }
          }
 
          fs_reg offset_reg;
@@ -4248,18 +4285,28 @@ fs_visitor::nir_emit_intrinsic(const fs_builder &bld, nir_intrinsic_instr *instr
                     brw_imm_ud(type_size * first_component));
          }
 
-         if (type_size == 2) {
+         if (type_size == 2 && length == 1) {
             /* Untyped Surface messages have a fixed 32-bit size, so we need
-             * to rely on byte scattered in order to write 16-bit elements.
+             * to rely on byte scattered in order to write a single 16-bit
+             * element.
              */
             emit_byte_scattered_write(bld, surf_index, offset_reg,
-                                      offset(val_reg, bld, first_component * type_slots),
+                                      current_val_reg,
                                       1 /* dims */, length * type_slots,
                                       BRW_PREDICATE_NONE);
          } else {
+            unsigned write_size = length * type_slots;
+
+            /* We need to half the write size of the untyped write message when
+             * submitting two packed 16-bit values, because length is 2 but the
+             * type size is 16-bit. So we are effectively writing just 32 bits
+             * (size = 1).
+             */
+            if (type_size == 2)
+               write_size /= 2;
             emit_untyped_write(bld, surf_index, offset_reg,
-                               offset(val_reg, bld, first_component * type_slots),
-                               1 /* dims */, length * type_slots,
+                               current_val_reg,
+                               1 /* dims */, write_size,
                                BRW_PREDICATE_NONE);
          }
 
-- 
2.11.0