[vmwgfx] crash upon vmw_open_channel() when booting on Oracle VM VirtualBox

[Tetsuo Handa]

On 2023/01/29 15:00, Zack Rusin wrote:
> On Sat, 2023-01-28 at 20:44 +0900, Tetsuo Handa wrote:
>> Hello.
>>
>> I noticed that a kernel built with vmwgfx driver fails to boot a Linux guest
>> on Oracle VM VirtualBox 7.0.4 on Windows 11 on DELL Inspiron 14 5420.
>> I didn't notice this problem when I booted an older kernel on an older version
>> of Oracle VM VirtualBox on Windows 8.1 on an older PC.
>>
>> The location that crashes is
>>
>>         VMW_PORT(VMW_PORT_CMD_OPEN_CHANNEL,
>>                 (protocol | GUESTMSG_FLAG_COOKIE), si, di,
>>                 0,
>>                 VMW_HYPERVISOR_MAGIC,
>>                 eax, ebx, ecx, edx, si, di);
>>
>> in vmw_open_channel(). It might be that some changes in VirtualBox side
>> is conflicting with how VMware tries to test if the guest is VMware.
>> How can I debug this problem?
> 
> You'd have to figure out what exactly is the problem. I couldn't reproduce it on
> vmware hypervisors with your .config.

This problem happens on not VMware hypervisors but VirtualBox hypervisors. 

>                                       FWIW that code has been there and hasn't been
> changed in years. Oracle emulated svga device always had problems, was never
> supported by vmwgfx and afaict is not maintained by Oracle so I'd strongly suggest
> that you switch to some other graphics device on virtualbox.

Indeed, not selecting VMSVGA as Graphics Controller in the screen tab of display
setting allowed me to boot the guest.

The reason I built-in vmwgfx is that I want to reuse the same kernel configuration on
multiple environments; syzbot uses a large kernel configuration that builts-in almost
everything.

> 
> In the meantime I think the attached patch should at least get you booting. You can
> give it a try and if it works I can push it sometime this week.

Yes, this patch allowed me to boot the guest when selecting VMSVGA as Graphics Controller.

Thank you.