Mesa (main): radeonsi/vcn: use calculated max hierarchy depth for hevc enc

[GitLab Mirror]

Module: Mesa
Branch: main
Commit: 3962555db89a12ecffd0d6a6259d96b24c60779e
URL:    http://cgit.freedesktop.org/mesa/mesa/commit/?id=3962555db89a12ecffd0d6a6259d96b24c60779e

Author: Boyuan Zhang <boyuan.zhang at amd.com>
Date:   Tue Jul 12 12:11:41 2022 -0400

radeonsi/vcn: use calculated max hierarchy depth for hevc enc

Certain player has hard coded max_transform_hierarchy_depth_inter and
max_transform_hierarchy_depth_intra values set through VA-API, which
doesn't work on radeon HW. Until properly fixing it on player side,
temporarily adding this workaround to use calculated values instead.

Signed-off-by: Boyuan Zhang <boyuan.zhang at amd.com>
Reviewed-by: Ruijing Dong <ruijing.dong at amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17489>

---

 src/gallium/drivers/radeonsi/radeon_vcn_enc.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/src/gallium/drivers/radeonsi/radeon_vcn_enc.c b/src/gallium/drivers/radeonsi/radeon_vcn_enc.c
index 0ae9b9ce881..0e608adf721 100644
--- a/src/gallium/drivers/radeonsi/radeon_vcn_enc.c
+++ b/src/gallium/drivers/radeonsi/radeon_vcn_enc.c
@@ -142,10 +142,17 @@ static void radeon_vcn_enc_get_param(struct radeon_encoder *enc, struct pipe_pic
          pic->seq.log2_min_transform_block_size_minus2;
       enc->enc_pic.log2_diff_max_min_transform_block_size =
          pic->seq.log2_diff_max_min_transform_block_size;
+
+      /* To fix incorrect hardcoded values set by player
+       * log2_diff_max_min_luma_coding_block_size = log2(64) - (log2_min_luma_coding_block_size_minus3 + 3)
+       * max_transform_hierarchy_depth_inter = log2_diff_max_min_luma_coding_block_size + 1
+       * max_transform_hierarchy_depth_intra = log2_diff_max_min_luma_coding_block_size + 1
+       */
       enc->enc_pic.max_transform_hierarchy_depth_inter =
-         pic->seq.max_transform_hierarchy_depth_inter;
+         6 - (pic->seq.log2_min_luma_coding_block_size_minus3 + 3) + 1;
       enc->enc_pic.max_transform_hierarchy_depth_intra =
-         pic->seq.max_transform_hierarchy_depth_intra;
+         enc->enc_pic.max_transform_hierarchy_depth_inter;
+
       enc->enc_pic.log2_parallel_merge_level_minus2 = pic->pic.log2_parallel_merge_level_minus2;
       enc->enc_pic.bit_depth_luma_minus8 = pic->seq.bit_depth_luma_minus8;
       enc->enc_pic.bit_depth_chroma_minus8 = pic->seq.bit_depth_chroma_minus8;