[PATCH] drm/amd/display: Fix 64 bit wraparound resulting in illegal drm mode

[Leslie Shi]

[Why]
For m = drm_display_mode{.clock = 533250, .htotal = 4000, .vtotal = 2222},
common_rates[i] = 60000, the result of target_vtotal is 2221. This cause
wraparound of variable target_vtotal_diff.

[How]
Skip the loop if target_vtotal less than m->vtotal

Signed-off-by: Leslie Shi <Yuliang.Shi at amd.com>
---
 drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
index c2bc7db85d7e..ed7ab089b224 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
@@ -8537,6 +8537,10 @@ static uint add_fs_modes(struct amdgpu_dm_connector *aconnector)
 		num = (unsigned long long)m->clock * 1000 * 1000;
 		den = common_rates[i] * (unsigned long long)m->htotal;
 		target_vtotal = div_u64(num, den);
+
+		if (target_vtotal < m->vtotal)
+			continue;
+
 		target_vtotal_diff = target_vtotal - m->vtotal;
 
 		/* Check for illegal modes */
-- 
2.25.1