[PATCH] drm/amd/display: Fix 64 bit wraparound resulting in illegal drm mode

[Shi, Leslie]

[AMD Official Use Only - General]

Please skip this patch, the description is incorrect. I will provide a new patch soon.

-----Original Message-----
From: Shi, Leslie <Yuliang.Shi at amd.com> 
Sent: Friday, June 17, 2022 4:56 PM
To: amd-gfx at lists.freedesktop.org; Deucher, Alexander <Alexander.Deucher at amd.com>
Cc: Chen, Guchun <Guchun.Chen at amd.com>; Shi, Leslie <Yuliang.Shi at amd.com>
Subject: [PATCH] drm/amd/display: Fix 64 bit wraparound resulting in illegal drm mode

[Why]
For m = drm_display_mode{.clock = 533250, .htotal = 4000, .vtotal = 2222}, common_rates[i] = 60000, the result of target_vtotal is 2221. This cause wraparound of variable target_vtotal_diff.

On RHEL7.9 kernel, the type of vsync_start is int and the value of new calculated vsync_start equals 67698. This lead to the following call
trace:

WARNING: CPU: 9 PID: 424 at drivers/gpu/drm/drm_modes.c:1642 drm_mode_convert_to_umode+0x42/0x1c0 [drm] timing values too large for mode info
CPU: 9 PID: 424 Comm: plymouthd Tainted: G           OE  ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: HP HP Z8 G4 Workstation/81C7, BIOS P60 v02.73 06/16/2021 Call Trace:
[<ffffffff93981340>] dump_stack+0x19/0x1b [<ffffffff9329b228>] __warn+0xd8/0x100 [<ffffffff9329b2af>] warn_slowpath_fmt+0x5f/0x80 [<ffffffffc041b4e2>] ? __drm_mode_object_find+0x72/0x100 [drm] [<ffffffffc040b622>] drm_mode_convert_to_umode+0x42/0x1c0 [drm] [<ffffffffc041a2d6>] drm_mode_getconnector+0x3c6/0x560 [drm] [<ffffffffc0419f10>] ? drm_connector_property_set_ioctl+0x60/0x60 [drm] [<ffffffffc040392c>] drm_ioctl_kernel+0xbc/0x100 [drm] [<ffffffffc0403bb4>] drm_ioctl+0x244/0x450 [drm] [<ffffffffc0419f10>] ? drm_connector_property_set_ioctl+0x60/0x60 [drm] [<ffffffffc051b30e>] amdgpu_drm_ioctl+0x4e/0x80 [amdgpu] [<ffffffff934632e0>] do_vfs_ioctl+0x3a0/0x5b0 [<ffffffff9398e678>] ? __do_page_fault+0x238/0x500 [<ffffffff93463591>] SyS_ioctl+0xa1/0xc0 [<ffffffff93993f92>] system_call_fastpath+0x25/0x2a

[How]
Skip the loop if target_vtotal less than m->vtotal

Signed-off-by: Leslie Shi <Yuliang.Shi at amd.com>
---
 drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
index c2bc7db85d7e..ed7ab089b224 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
@@ -8537,6 +8537,10 @@ static uint add_fs_modes(struct amdgpu_dm_connector *aconnector)
 		num = (unsigned long long)m->clock * 1000 * 1000;
 		den = common_rates[i] * (unsigned long long)m->htotal;
 		target_vtotal = div_u64(num, den);
+
+		if (target_vtotal < m->vtotal)
+			continue;
+
 		target_vtotal_diff = target_vtotal - m->vtotal;
 
 		/* Check for illegal modes */
--
2.25.1