[PATCH] drm/amd/display: Fix 64 bit wraparound resulting in illegal drm mode

[Leslie Shi]

[Why]
For m = drm_display_mode{.clock = 533250, .htotal = 4000, .vtotal = 2222},
common_rates[i] = 60000, the result of target_vtotal is 2221. This cause
wraparound of variable target_vtotal_diff.

On RHEL7.9 kernel, the type of vsync_start is int and the value of new
calculated vsync_start equals 67698. This lead to the following call
trace:

WARNING: CPU: 9 PID: 424 at drivers/gpu/drm/drm_modes.c:1642 drm_mode_convert_to_umode+0x42/0x1c0 [drm]
timing values too large for mode info
CPU: 9 PID: 424 Comm: plymouthd Tainted: G           OE  ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: HP HP Z8 G4 Workstation/81C7, BIOS P60 v02.73 06/16/2021
Call Trace:
[<ffffffff93981340>] dump_stack+0x19/0x1b
[<ffffffff9329b228>] __warn+0xd8/0x100
[<ffffffff9329b2af>] warn_slowpath_fmt+0x5f/0x80
[<ffffffffc041b4e2>] ? __drm_mode_object_find+0x72/0x100 [drm]
[<ffffffffc040b622>] drm_mode_convert_to_umode+0x42/0x1c0 [drm]
[<ffffffffc041a2d6>] drm_mode_getconnector+0x3c6/0x560 [drm]
[<ffffffffc0419f10>] ? drm_connector_property_set_ioctl+0x60/0x60 [drm]
[<ffffffffc040392c>] drm_ioctl_kernel+0xbc/0x100 [drm]
[<ffffffffc0403bb4>] drm_ioctl+0x244/0x450 [drm]
[<ffffffffc0419f10>] ? drm_connector_property_set_ioctl+0x60/0x60 [drm]
[<ffffffffc051b30e>] amdgpu_drm_ioctl+0x4e/0x80 [amdgpu]
[<ffffffff934632e0>] do_vfs_ioctl+0x3a0/0x5b0
[<ffffffff9398e678>] ? __do_page_fault+0x238/0x500
[<ffffffff93463591>] SyS_ioctl+0xa1/0xc0
[<ffffffff93993f92>] system_call_fastpath+0x25/0x2a

[How]
Skip the loop if target_vtotal less than m->vtotal

Signed-off-by: Leslie Shi <Yuliang.Shi at amd.com>
---
 drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
index c2bc7db85d7e..ed7ab089b224 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
@@ -8537,6 +8537,10 @@ static uint add_fs_modes(struct amdgpu_dm_connector *aconnector)
 		num = (unsigned long long)m->clock * 1000 * 1000;
 		den = common_rates[i] * (unsigned long long)m->htotal;
 		target_vtotal = div_u64(num, den);
+
+		if (target_vtotal < m->vtotal)
+			continue;
+
 		target_vtotal_diff = target_vtotal - m->vtotal;
 
 		/* Check for illegal modes */
-- 
2.25.1