[PATCH] drm/vmwgfx: Clamp the max COTable size

[Zack Rusin]

On Mon, Jul 28, 2025 at 5:08 PM Ian Forbes <ian.forbes at broadcom.com> wrote:
>
> Due to a quirk of the SVGA device the MOB that backs the COTable cannot
> exceed SVGA_COTABLE_MAX_IDS elements. This causes issues because MOBs
> are allocated in PAGE_SIZE chunks which means we always round up the
> size to the next page. This causes the device to error out when we try
> to set a COTable that can store SVGA_COTABLE_MAX_IDS but is slightly
> over-allocated by some fraction of a page. This then leaves the device
> in an indeterminate state.
>
> Due to the doubling logic in vmw_cotable_create this occurs when we
> allocate slightly more than half the available IDs. As a result the
> validation logic in vmw_cotable_notify never trips which is what we
> currently rely on to limit the size of the COTable indirectly.
>
> By reducing SVGA_COTABLE_MAX_IDS to a slightly smaller value we can avoid
> this quirk. Any application getting close to this limit is likely
> leaking resources so reducing the size by less than 1% should not have
> any noticeable impact on well behaving applications.
>
> Fixes: d80efd5cb3de ("drm/vmwgfx: Initial DX support")
> Signed-off-by: Ian Forbes <ian.forbes at broadcom.com>
> ---
>  drivers/gpu/drm/vmwgfx/vmwgfx_cotable.c | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_cotable.c b/drivers/gpu/drm/vmwgfx/vmwgfx_cotable.c
> index 98331c4c0335..366e97a95179 100644
> --- a/drivers/gpu/drm/vmwgfx/vmwgfx_cotable.c
> +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_cotable.c
> @@ -74,6 +74,11 @@ struct vmw_cotable_info {
>                             bool);
>  };
>
> +/*
> + * Due to a quirk of SVGA device we can't actually allocate SVGA_COTABLE_MAX_IDS
> + * for all resource types. This new limit will work regardless of type.
> + */
> +#define SVGA_COTABLE_EFFECTIVE_MAX_IDS (SVGA_COTABLE_MAX_IDS - 510)
>
>  /*
>   * Getting the initial size right is difficult because it all depends
> @@ -545,6 +550,7 @@ static int vmw_cotable_create(struct vmw_resource *res)
>  {
>         struct vmw_cotable *vcotbl = vmw_cotable(res);
>         size_t new_size = res->guest_memory_size;
> +       size_t max_size = co_info[vcotbl->type].size * SVGA_COTABLE_EFFECTIVE_MAX_IDS;
>         size_t needed_size;
>         int ret;

I'm not sure about this one. I suspect that what we really want is to
just properly align new_size to a page size. I think we do it (align
sizes to a page size) in a bunch of places. In particular the buffer
that's actually backing the cotable will be page aligned so if you
replace res->guest_memory_size with
res->guest_memory_bo->tbo.base.size it might fix it. But in general
the issue is that res->guest_memory_size is pretty severely broken,
there's a bunch of places in vmwgfx_resource.c that uses it as a
boolean.

z
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5414 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://lists.freedesktop.org/archives/dri-devel/attachments/20250728/f26e78ae/attachment-0001.bin>