[REGRESSION] drm/vmwgfx shows green Xv instead of actual video (bisected I think)

[Doug Brown]

Hello,

I have discovered a problem in vmwgfx that caused Xv playback to break
between v6.3 and v6.4-rc1. In v6.3, inside of an Ubuntu 24.04 VMware VM
with 3D acceleration disabled, I can run the following GStreamer command
("sudo apt install gstreamer1.0-tools" if you don't already have it):

gst-launch-1.0 videotestsrc ! 
video/x-raw,format=YV12,width=640,height=480 ! xvimagesink

And it works fine, showing a test pattern.

In v6.4-rc1 and all the way up to a build I just made today of Linus's
latest master branch and also including Zack's latest patchset [1]:

Linux doug-Ubuntu-MATE-2404 6.10.0-rc7+ #55 SMP PREEMPT_DYNAMIC Thu Jul
11 12:46:06 PDT 2024 x86_64 x86_64 x86_64 GNU/Linux

...if I run the same command, the GStreamer window shows up containing
solid green rather than the test pattern.

I believe I have bisected the problem to commit:

39985eea5a6d ("drm/vmwgfx: Abstract placement selection")

However, the bisect process was complicated because two earlier commits
temporarily broke vmwgfx so I had to undo them during bisection, which
also required some conflict resolution:

180253782038 ("drm/ttm: stop allocating dummy resources during BO 
creation") and
f87c1f0b7b79 ("drm/ttm: prevent moving of pinned BOs")

Note that if you have gstreamer1.0-vaapi installed, you should
temporarily remove it first before testing because there is a separate
Xorg bug that causes Xorg to crash when you run the above command if
gstreamer1.0-vaapi is installed.

I'm happy to do any further testing as needed. I've reproduced this
problem across several different machines and distros. A fully-updated
Arch Linux is also affected.

As a side note, that same gst-launch-1.0 command is also currently
broken with 3D acceleration enabled, but that's a separate issue in mesa
that I've submitted a patch for [2].

Thanks,
Doug

[1] 
https://lore.kernel.org/all/20240702021254.1610188-1-zack.rusin@broadcom.com/#t
[2] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30116

#regzbot introduced: v6.3..v6.4-rc1