[PATCH] drm/vmwgfx: Clamp the max COTable size

[Ian Forbes]

Due to a quirk of the SVGA device the MOB that backs the COTable cannot
exceed SVGA_COTABLE_MAX_IDS elements. This causes issues because MOBs
are allocated in PAGE_SIZE chunks which means we always round up the
size to the next page. This causes the device to error out when we try
to set a COTable that can store SVGA_COTABLE_MAX_IDS but is slightly
over-allocated by some fraction of a page. This then leaves the device
in an indeterminate state.

Due to the doubling logic in vmw_cotable_create this occurs when we
allocate slightly more than half the available IDs. As a result the
validation logic in vmw_cotable_notify never trips which is what we
currently rely on to limit the size of the COTable indirectly.

By reducing SVGA_COTABLE_MAX_IDS to a slightly smaller value we can avoid
this quirk. Any application getting close to this limit is likely
leaking resources so reducing the size by less than 1% should not have
any noticeable impact on well behaving applications.

Fixes: d80efd5cb3de ("drm/vmwgfx: Initial DX support")
Signed-off-by: Ian Forbes <ian.forbes at broadcom.com>
---
 drivers/gpu/drm/vmwgfx/vmwgfx_cotable.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_cotable.c b/drivers/gpu/drm/vmwgfx/vmwgfx_cotable.c
index 98331c4c0335..366e97a95179 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_cotable.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_cotable.c
@@ -74,6 +74,11 @@ struct vmw_cotable_info {
 			    bool);
 };
 
+/*
+ * Due to a quirk of SVGA device we can't actually allocate SVGA_COTABLE_MAX_IDS
+ * for all resource types. This new limit will work regardless of type.
+ */
+#define SVGA_COTABLE_EFFECTIVE_MAX_IDS (SVGA_COTABLE_MAX_IDS - 510)
 
 /*
  * Getting the initial size right is difficult because it all depends
@@ -545,6 +550,7 @@ static int vmw_cotable_create(struct vmw_resource *res)
 {
 	struct vmw_cotable *vcotbl = vmw_cotable(res);
 	size_t new_size = res->guest_memory_size;
+	size_t max_size = co_info[vcotbl->type].size * SVGA_COTABLE_EFFECTIVE_MAX_IDS;
 	size_t needed_size;
 	int ret;
 
@@ -553,6 +559,8 @@ static int vmw_cotable_create(struct vmw_resource *res)
 	while (needed_size > new_size)
 		new_size *= 2;
 
+	new_size = MIN(new_size, max_size);
+
 	if (likely(new_size <= res->guest_memory_size)) {
 		if (vcotbl->scrubbed && vmw_resource_mob_attached(res)) {
 			ret = vmw_cotable_unscrub(res);
@@ -650,7 +658,7 @@ int vmw_cotable_notify(struct vmw_resource *res, int id)
 {
 	struct vmw_cotable *vcotbl = vmw_cotable(res);
 
-	if (id < 0 || id >= SVGA_COTABLE_MAX_IDS) {
+	if (id < 0 || id >= SVGA_COTABLE_EFFECTIVE_MAX_IDS) {
 		DRM_ERROR("Illegal COTable id. Type is %u. Id is %d\n",
 			  (unsigned) vcotbl->type, id);
 		return -EINVAL;
-- 
2.50.1