[PATCH v2] drm/vmwgfx: Filter modes which exceed 3/4 of graphics memory.

[Zack Rusin]

On Tue, Feb 6, 2024 at 4:30 PM Ian Forbes <ian.forbes at broadcom.com> wrote:
>
> So the issue is that SVGA_3D_CMD_DX_PRED_COPY_REGION between 2
> surfaces that are the size of the mode fails. Technically for this to
> work the filter will have to be 1/2 of graphics mem. I was just lucky
> that the next mode in the list was already less than 1/2. 3/4 is not
> actually going to work. Also this only happens on X/Gnome and seems
> more like an issue with the compositor. Wayland/Gnome displays the
> desktop but it's unusable and glitches even with the 1/2 limit. I
> don't think wayland even abides by the mode limits as I see it trying
> to create surfaces larger than the mode. It might be using texture
> limits instead.

So the SVGA_3D_CMD_DX_PRED_COPY_REGION is only available with dx
contexts/3d enabled/gb surfaces. With 3d or gb objects disabled we
should fall back to legacy display and that command shouldn't have
been used. Is that the case? Does it work with 3d/gb objects disabled?

There's a few bugs there:
- SVGA_3D_CMD_DX_PRED_COPY_REGION should only come from userspace, the
userspace should validate that the max amount of resident memory
hasn't been exceeded before issuing those copies
- vmwgfx should be a lot better about determining whether the amount
of resident memory required by the current command buffers hasn't been
exceeded
- In case of high memory pressure vmwgfx should explicitly disable 3d
support. There's no way to run 3d workloads with anything less than
64mb of ram especially given that we do not adjust our texture limits
and they will remain either 4k, 8k or more depending on what we're
running on.

But those are secondary to making resolution switch work correctly on
basic system, i.e.:
1) Disable 3D and gb objects
2) Check if in the kernel log vmwgx says that it's using "legacy display"
3) Check if the resolution switching works correctly
4) If not lets fix that first (fix #1)
5) Disable 3D and keep gb objects active
6) Check that the kernel log select "screen target display unit" and
have 3d disabled (i.e. no SVGA_3D_CMD_DX_PRED_COPY_REGION is coming
through)
7) If that doesn't work lets fix that next (fix #2)
8) Enabled 3d and gb objects (your current default)
9) Check if max_mob_pages (i.e. max_resident_memory) is smaller than
what we'd need to hold even a single a texture limits * 4bpp, print a
warning and disable 3d (this should bring us in line with what we
fixed in point #7) (fix #3)

So basically we want to make sure that on vmwgfx all three
configurations work: 1) 3d and gb objects disabled, 2) 3d disabled, gb
objects enabled, 3) 3d and gb object enabled.

z