GPU passthrough support for Stoney [Radeon R2/R3/R4/R5 Graphics]?

Wed May 22 23:00:03 UTC 2019

On Wed, May 22, 2019 at 1:39 PM Alex Deucher <alexdeucher at gmail.com> wrote:
>
> On Tue, May 21, 2019 at 1:46 PM Micah Morton <mortonm at chromium.org> wrote:
> >
> > On Fri, May 17, 2019 at 9:59 AM Alex Deucher <alexdeucher at gmail.com> wrote:
> > >
> > > On Fri, May 17, 2019 at 11:36 AM Micah Morton <mortonm at chromium.org> wrote:
> > > >
> > > > On Thu, May 16, 2019 at 1:39 PM Alex Deucher <alexdeucher at gmail.com> wrote:
> > > > >
> > > > > On Thu, May 16, 2019 at 4:07 PM Micah Morton <mortonm at chromium.org> wrote:
> > > > > >
> > > > > > On Wed, May 15, 2019 at 7:19 PM Alex Deucher <alexdeucher at gmail.com> wrote:
> > > > > > >
> > > > > > > On Wed, May 15, 2019 at 2:26 PM Micah Morton <mortonm at chromium.org> wrote:
> > > > > > > >
> > > > > > > > Hi folks,
> > > > > > > >
> > > > > > > > I'm interested in running a VM on a system with an integrated Stoney
> > > > > > > > [Radeon R2/R3/R4/R5 Graphics] card and passing through the graphics
> > > > > > > > card to the VM using the IOMMU. I'm wondering whether this is feasible
> > > > > > > > and supposed to be doable with the right setup (as opposed to passing
> > > > > > > > a discrete GPU to the VM, which I think is definitely doable?).
> > > > > > > >
> > > > > > > > So far, I can do all the qemu/kvm/vfio/iommu stuff to run the VM and
> > > > > > > > pass the integrated GPU to it, but the drm driver in the VM fails
> > > > > > > > during amdgpu_device_init(). Specifically, the logs show the SMU being
> > > > > > > > unresponsive, which leads to a 'SMU firmware load failed' error
> > > > > > > > message and kernel panic. I can share VM logs and the invocation of
> > > > > > > > qemu and such if helpful, but first wanted to know at a high level if
> > > > > > > > this should be feasible?
> > > > > > > >
> > > > > > > > P.S.: I'm not initializing the GPU in the host bios or host kernel at
> > > > > > > > all, so I should be passing a fresh GPU to the VM. Also, I'm pretty
> > > > > > > > sure I'm running the correct VGA bios for this GPU in the guest VM
> > > > > > > > bios before guest boot.
> > > > > > > >
> > > > > > > > Any comments/suggestions would be appreciated!
> > > > > > >
> > > > > > > It should work in at least once as long as your vm is properly set up.
> > > > > >
> > > > > > Is there any reason running coreboot vs UEFI at host boot would make a
> > > > > > difference? I was running a modified version of coreboot that avoids
> > > > > > doing any GPU initialization in firmware -- so the first POST happens
> > > > > > inside the guest.
> > > > >
> > > > > The GPU on APUs shares a bunch of resources with the CPU.  There are a
> > > > > bunch of blocks which are shared and need to be initialized on both
> > > > > for everything to work properly.
> > > >
> > > > Interesting. So skipping running the vbios in the host and waiting
> > > > until running it for the first time in the guest SeaBIOS is a bad
> > > > idea? Would it be better to let APU+CPU initialize normally in the
> > > > host and then skip trying to run the vbios in guest SeaBIOS and just
> > > > do some kind of reset before the drm driver starts accessing it from
> > > > the guest?
> > >
> > > If you let the sbios initialize things, it should work.  The driver
> > > will do the right thing to init the card when it loads whether its
> > > running on bare metal or in a VM.  We've never tested any scenarios
> > > where the GPU on APUs is not handled by the sbios.  Note that the GPU
> > > does not have to be posted per se, it just needs to have been properly
> > > taken into account when the sbios comes up so that shared components
> > > are initialized correctly.  I don't know what your patched system does
> > > or doesn't do with respect to the platform initialization.
> >
> > So it sounds like you are suggesting the following:
> >
> > a) Run the vbios as part of the host sbios to initialize stuff and
> > patch up the vbios
> >
> > b) Don't run the drm/amdgpu driver in the host, wait until the guest for that
> >
> > c) Avoid running the vbios (again) in the guest sbios since it has
> > already been run. Although since "the driver needs access to the vbios
> > image in the guest to get device specific configuration details" I
> > should still do '-device
> > vfio-pci,...,romfile=/path/to/vbios-that-was-fixed-up-by-host-sbios',
> > but should patch the guest sbios to avoid running the vbios again in
> > the guest?
> >
> > d) run the drm/amdgpu driver in the guest on the hardware that was
> > initialized in the host sbios
> >
> > Am I getting this right?
> >
>
> I would suggest starting with a standard sbios/vbios.  Boot the system
> as usual.  You can blacklist amdgpu if you don't want gpu driver
> loaded on the host.  Get a copy of the vbios image.  Depending on the
> platform, it may be available via the standard vbios shadow location
> or it may be at the start of carveout or it may be available via an
> ACPI interface.  The driver has the logic to fetch it.  You can dump a
> copy from the driver via /sys/kernel/debug/dri/X/amdgpu_vbios where X
> is the number of the dri device of the card in question.  Then start

I've been able to retrieve the vbios from
/sys/kernel/debug/dri/X/amdgpu_vbios as well as
/sys/devices/pciXXXX:XX/XXXX:XX:XX.X/rom (although they do differ in
46 different byte locations, not sure if this is expected). In both
cases, passing the vbios through to the VM results in SeaBIOS hanging
while executing the vbios:

Found 2 cpu(s) max supported 2 cpu(s)
Copying PIR from 0x7ffbfc60 to 0x000f5e00
Copying MPTABLE from 0x00006e60/7ffa34c0 to 0x000f5cf0
Copying SMBIOS entry point from 0x00006e60 to 0x000f5b10
Scan for VGA option rom
Running option rom at c000:0003
HANGS HERE INDEFINITELY <--

Which is why I was asking about not running the vbios in the guest
SeaBIOS. Granted, the sbios I've been running in the host is Chrome OS
coreboot firmware -- so probably not what you mean by "standard
sbios"? You think there is a good chance running coreboot instead of
the "standard sbios" is what is causing things to fail?

> VM with the device passed through.  Don't worry about running the
> vbios or not in the host vs. the guest.  The driver will do the right
> thing.
>
> Alex
>
> > >
> > > >
> > > > >
> > > > > >
> > > > > > > Note that the driver needs access to the vbios image in the guest to
> > > > > > > get device specific configuration details (clocks, display connector
> > > > > > > configuration, etc.).
> > > > > >
> > > > > > Is there anything I need to do to ensure this besides passing '-device
> > > > > > vfio-pci,...,romfile=/path/to/vgarom' to qemu?
> > > > >
> > > > > You need the actual vbios rom image from your system.  The image is
> > > > > board specific.
> > > >
> > > > I should have the correct vbios rom image for my board. I'm extracting
> > > > it from the firmware image (that works for regular graphics init
> > > > without this VM stuff) for the board at build time (rather than
> > > > grabbing it from /sys/devices/pci... at runtime), so it shouldn't be
> > > > modified or corrupted in any way.
> > >
> > > The vbios image is patched at boot time by the sbios image for run
> > > time configuration stuff.  For example, some of the pcie lanes are
> > > shared with display lanes and can be used for either display or pcie
> > > add in cards.  The sbios determines this at boot and patches the vbios
> > > display tables so the driver knows that the displays are not
> > > available.  Also things like flat panels on laptops.  OEMs may have
> > > several different flat panel models they use with a particular
> > > platform and the sbios patches the vbios display tables with the
> > > proper parameters for the panel in use.  The sbios also patches tables
> > > related to bandwidth.  E.g., the type and speed and number of channels
> > > of the system ram so that the GPU driver can set proper limits on
> > > things like display modes.  So you need to use the vbios image that is
> > > provided by the sbios at boot.
> >
> > Ok yeah good point. I noticed that there are a few hundred byte
> > locations in the vbios that are different between the pre-boot version
> > from build time and the version dumped from sysfs at runtime, so
> > something must be getting patched up.
> >
> > >
> > > Alex