[vmwgfx] crash upon vmw_open_channel() when booting on Oracle VM VirtualBox

[Zack Rusin]

On Sun, 2023-01-29 at 17:11 +0900, Tetsuo Handa wrote:
> On 2023/01/29 15:00, Zack Rusin wrote:
> > On Sat, 2023-01-28 at 20:44 +0900, Tetsuo Handa wrote:
> > > Hello.
> > > 
> > > I noticed that a kernel built with vmwgfx driver fails to boot a Linux guest
> > > on Oracle VM VirtualBox 7.0.4 on Windows 11 on DELL Inspiron 14 5420.
> > > I didn't notice this problem when I booted an older kernel on an older version
> > > of Oracle VM VirtualBox on Windows 8.1 on an older PC.
> > > 
> > > The location that crashes is
> > > 
> > >         VMW_PORT(VMW_PORT_CMD_OPEN_CHANNEL,
> > >                 (protocol | GUESTMSG_FLAG_COOKIE), si, di,
> > >                 0,
> > >                 VMW_HYPERVISOR_MAGIC,
> > >                 eax, ebx, ecx, edx, si, di);
> > > 
> > > in vmw_open_channel(). It might be that some changes in VirtualBox side
> > > is conflicting with how VMware tries to test if the guest is VMware.
> > > How can I debug this problem?
> > 
> > You'd have to figure out what exactly is the problem. I couldn't reproduce it on
> > vmware hypervisors with your .config.
> 
> This problem happens on not VMware hypervisors but VirtualBox hypervisors. 

Yea, that's why I can't debug it myself and would need someone else to figure out
the exact issue.

> 
> >                                       FWIW that code has been there and hasn't
> > been
> > changed in years. Oracle emulated svga device always had problems, was never
> > supported by vmwgfx and afaict is not maintained by Oracle so I'd strongly
> > suggest
> > that you switch to some other graphics device on virtualbox.
> 
> Indeed, not selecting VMSVGA as Graphics Controller in the screen tab of display
> setting allowed me to boot the guest.
> 
> The reason I built-in vmwgfx is that I want to reuse the same kernel configuration
> on
> multiple environments; syzbot uses a large kernel configuration that builts-in
> almost
> everything.

Building vmwgfx is great, it's selecting the vmsvga device in virtualbox that's the
problem. afaict that device is unmaintained and unsupported.


> > In the meantime I think the attached patch should at least get you booting. You
> > can
> > give it a try and if it works I can push it sometime this week.
> 
> Yes, this patch allowed me to boot the guest when selecting VMSVGA as Graphics
> Controller.
> 
> Thank you.

Great. I'll clean it up and send it for review to dri-devel sometime this week.

z