A query on frame buffers
Konrad Rzeszutek Wilk
konrad.wilk at oracle.com
Mon Jan 10 12:24:51 PST 2011
> AMD chipset.
They don't seem to have have any errata's for this GFX business.
But they do have a passthrough mode - hopefull that is not what you have
passed in?
> > Since you mentioned that you were able to passthrough other PCI devices
> > that means the IOMMU is working. This narrows it down.
>
> Yes I could pass-through a network card to VM. I also tried passing
> through a old Sound Card to Windows 7 machine. It did not work as
> expected, Windows 7 does not have the old drivers.
Heh.
>
> >
> > The big difference between other PCI devices and the graphics card
> > is that the graphic card has its own VMM and "radeon_gart_bind" ends
> > up programming the bus address (so virt_to_phys of the pages, and
> > you could put in a printk there to see what it is) in the graphic
> > VMM. This means that when the graphic card is told to fetch the
> > writeback data it ends up using the address that was programmed
> > in there to look. Here the IOMMU should step in and see "oh, this
> > DMA address is for this guest, let me patch it up and actually
> > look where this would fall in the guest space. Oh OK, let me use
> > this value real value which correspond the guests real value."
>
> It will take some time for me to understand every thing you have written :)
Ok. Ping me if you have some questions. Reading this might
help (or confuse you more): http://wiki.xensource.com/xenwiki/XenPVOPSDRM
>
> >
> > But the IOMMU has a couple of knobs. One of those is the passthrough
> > code and the GFX mapping code. The first should be easy to spot (should
> > tell you on bootup whether you got that enabled or not), while the other
> > looks to be a bunch of workarounds when passing in GFX cards b/c they ..
> > well, not sure what, but they look to not work correctly with the IOMMU.
>
> By GFX Mapping,
> Do you mean the code that maps VGA frame-buffers to Guest OS?
No. It is the IOMMU kernel code that checks to see if this is a GFX
card and if so does something fancy. But the workarounds for this appear
to be exclusive to Intel chipset so you shouldn't hit any.
>
> One more silly question on the frame-buffer mapping.
>
> I was reading a paper on xen vga pass-though. It says "When applying
> PCI pass-through, certain memory areas of the physical machine are
> mapped to the VM. When the guest OS writes to one of those memory
> addresses, Xen will make sure it is actually written at the
> appropriate address. This implicates that no other VM is able to make
> use of that device. The frame-buffer is mapped to the machine's main
> memory at address 0xA0000 to 0xC0000. This memory range must be mapped
> to the VM's memory in order for the OS to address the video adapter."
<nods>
>
> My question is....
> In my case, the host machine is using Nvidia graphics card, so the
> host frame-buffer memory 0xA0000 to 0xC0000 is being actively used by
> host. Now if the ATI card maps it's framebuffer to this same address
> wouldn't it cause any problem.
Ah, but it does not. The first card that is called by the BIOS (so your
motherboard BIOS) sets itself up to use that memory. The other card has
to wait. Keep in mind that the A0000 to C0000 is the old style VGA
framebuffer (80x25) or if you program the card properly it can do
VESA graphics, but nothing else. Usually the guest gets a Cirrus Logic
video card passed in which sets this device up (and touching that
area in the guest ends up in VNC window or SDL window).
Anyhow back to your host..
If you grep in your kernel log you should see something like:
[drm] fb mappable at 0xE0142000
This is the framebuffer address.
>
> If the the machine has only 1 graphics card those statement make
> sense, but I could not understand what will happen when there are two
> graphics cards, each connected to different monitor and assigned to
> different VM, where would each machines frame-buffer reside in the
> host memory.
Check the kernel log. Usually it is also the first BAR in the pci device.
>
> Does Xen allow passing-though multiple graphics cards, each to different VM?
Yes (with some patches posted by the AMD and Intel folks)..
(look for threads that have some GFX or Radeon or something like that in its title).
>
> >
> > If you see 'identity map for device' where the device is your GFX
> > then that looks to be the bug. It shouldn't set the identity mapping on it.
> >
>
> I guess the identity map is used only on the Intel chipset. Here are
> msgs on host machine when addition debugging parameter
> (amd_iommu_dump) was added.
>
> prasad at prasad-kvm:~$ dmesg | grep -i -e iommu -e dmar -e amd
> [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+
> root=/dev/sda1 ro iommu=1 amd_iommu_dump
> [ 0.000000] ACPI: SRAT 00000000bfe9f8b0 00108 (v01 AMD FAM_F_10
> 00000002 AMD 00000001)
> [ 0.000000] ACPI: IVRS 00000000bfe9fa00 000D8 (v01 AMD RD890S
> 00202031 AMD 00000000)
> [ 0.000000] ACPI: SSDT 00000000bfe9fae0 00DA4 (v01 A M I POWERNOW
> 00000001 AMD 00000001)
> [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+
> root=/dev/sda1 ro iommu=1 amd_iommu_dump
> [ 0.000000] Please enable the IOMMU option in the BIOS setup
> [ 0.012918] Performance Events: AMD PMU driver.
> [ 0.137960] CPU0: AMD Phenom(tm) II X6 1090T Processor stepping 00
> [ 0.300011] System has AMD C1E enabled
> [ 2.680386] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300
> [ 2.680441] AMD-Vi: mmio-addr: 00000000f6000000
> [ 2.680670] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00
> [ 2.680726] AMD-Vi: DEV_RANGE_END devid: 00:00.2
> [ 2.680776] AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00
> [ 2.680827] AMD-Vi: DEV_SELECT devid: 07:00.0 flags: 00
> [ 2.680883] AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00
> [ 2.680934] AMD-Vi: DEV_SELECT devid: 06:00.0 flags: 00
> [ 2.680984] AMD-Vi: DEV_SELECT devid: 00:05.0 flags: 00
> [ 2.681035] AMD-Vi: DEV_SELECT devid: 05:00.0 flags: 00
> [ 2.681086] AMD-Vi: DEV_SELECT devid: 00:06.0 flags: 00
> [ 2.681137] AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00
> [ 2.681187] AMD-Vi: DEV_SELECT devid: 00:07.0 flags: 00
> [ 2.681238] AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00
> [ 2.681289] AMD-Vi: DEV_SELECT devid: 00:0b.0 flags: 00
> [ 2.681340] AMD-Vi: DEV_SELECT_RANGE_START devid: 02:00.0 flags: 00
> [ 2.681391] AMD-Vi: DEV_RANGE_END devid: 02:00.1
> [ 2.681441] AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00
> [ 2.681492] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00
> [ 2.681543] AMD-Vi: DEV_RANGE_END devid: 00:12.2
> [ 2.681594] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00
> [ 2.681645] AMD-Vi: DEV_RANGE_END devid: 00:13.2
> [ 2.681695] AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7
> [ 2.681746] AMD-Vi: DEV_SELECT devid: 00:14.1 flags: 00
> [ 2.681797] AMD-Vi: DEV_SELECT devid: 00:14.2 flags: 00
> [ 2.681847] AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00
> [ 2.681898] AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00
> [ 2.681949] AMD-Vi: DEV_ALIAS_RANGE devid: 01:00.0 flags: 00
> devid_to: 00:14.4
> [ 2.682010] AMD-Vi: DEV_RANGE_END devid: 01:1f.7
> [ 2.682855] AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00
> [ 2.682906] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00
> [ 2.682957] AMD-Vi: DEV_RANGE_END devid: 00:16.2
> [ 2.683234] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
> [ 2.692043] AMD-Vi: Lazy IO/TLB flushing enabled
> [ 4.360112] ehci_hcd 0000:00:12.2: applying AMD
> SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
> [ 4.380635] ehci_hcd 0000:00:13.2: applying AMD
> SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
> [ 4.400658] ehci_hcd 0000:00:16.2: applying AMD
> SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
> [ 4.811320] powernow-k8: Found 1 AMD Phenom(tm) II X6 1090T
> Processor (6 cpu cores) (version 2.20.00)
> [ 12.349919] EDAC amd64_edac: Ver: 3.3.0 Dec 21 2010
> [ 12.350018] EDAC amd64: This node reports that Memory ECC is
> currently disabled, set F3x44[22] (0000:00:18.3).
> [ 12.350030] EDAC amd64: ECC disabled in the BIOS or no ECC
> capability, module will not load.
> [ 12.350044] amd64_edac: probe of 0000:00:18.2 failed with error -22
What motherboard is this?
>
> prasad at prasad-kvm:~$ lspci | grep -i RV
> 02:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B64
> [FireGL V3100 (PCIE)] (rev 80)
> 02:00.1 Display controller: ATI Technologies Inc RV370 5B64 [FireGL
> V3100 (PCIE)] (Secondary) (rev 80)
So, how come you aren't passing in 02:00.1?
>
> > Look for 'DMAR GFX' on Google. Might shed some more light.
> >
>
> No DMAR for AMD. I think I should have a look at DMAR GFX.
>
> I am really thankful to you for your support.
Hopefully I haven't confused you any further :-)
>
> Thanks and Regards,
> Prasad
More information about the dri-devel
mailing list