[vmwgfx] crash upon vmw_open_channel() when booting on Oracle VM VirtualBox

[Zack Rusin]

On Sat, 2023-01-28 at 20:44 +0900, Tetsuo Handa wrote:
> Hello.
> 
> I noticed that a kernel built with vmwgfx driver fails to boot a Linux guest
> on Oracle VM VirtualBox 7.0.4 on Windows 11 on DELL Inspiron 14 5420.
> I didn't notice this problem when I booted an older kernel on an older version
> of Oracle VM VirtualBox on Windows 8.1 on an older PC.
> 
> The location that crashes is
> 
>         VMW_PORT(VMW_PORT_CMD_OPEN_CHANNEL,
>                 (protocol | GUESTMSG_FLAG_COOKIE), si, di,
>                 0,
>                 VMW_HYPERVISOR_MAGIC,
>                 eax, ebx, ecx, edx, si, di);
> 
> in vmw_open_channel(). It might be that some changes in VirtualBox side
> is conflicting with how VMware tries to test if the guest is VMware.
> How can I debug this problem?

You'd have to figure out what exactly is the problem. I couldn't reproduce it on
vmware hypervisors with your .config. FWIW that code has been there and hasn't been
changed in years. Oracle emulated svga device always had problems, was never
supported by vmwgfx and afaict is not maintained by Oracle so I'd strongly suggest
that you switch to some other graphics device on virtualbox.

In the meantime I think the attached patch should at least get you booting. You can
give it a try and if it works I can push it sometime this week.

z

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-drm-vmwgfx-Print-errors-when-running-on-broken-unsup.patch
Type: text/x-patch
Size: 3733 bytes
Desc: 0001-drm-vmwgfx-Print-errors-when-running-on-broken-unsup.patch
URL: <https://lists.freedesktop.org/archives/dri-devel/attachments/20230129/33948c60/attachment-0001.bin>