A query on frame buffers
Prasad Joshi
prasadjoshi124 at gmail.com
Tue Jan 11 08:58:47 PST 2011
On Mon, Jan 10, 2011 at 8:24 PM, Konrad Rzeszutek Wilk
<konrad.wilk at oracle.com> wrote:
>> AMD chipset.
>
> They don't seem to have have any errata's for this GFX business.
> But they do have a passthrough mode - hopefull that is not what you have
> passed in?
>
>> > Since you mentioned that you were able to passthrough other PCI devices
>> > that means the IOMMU is working. This narrows it down.
>>
>> Yes I could pass-through a network card to VM. I also tried passing
>> through a old Sound Card to Windows 7 machine. It did not work as
>> expected, Windows 7 does not have the old drivers.
>
> Heh.
>>
>> >
>> > The big difference between other PCI devices and the graphics card
>> > is that the graphic card has its own VMM and "radeon_gart_bind" ends
>> > up programming the bus address (so virt_to_phys of the pages, and
>> > you could put in a printk there to see what it is) in the graphic
>> > VMM. This means that when the graphic card is told to fetch the
>> > writeback data it ends up using the address that was programmed
>> > in there to look. Here the IOMMU should step in and see "oh, this
>> > DMA address is for this guest, let me patch it up and actually
>> > look where this would fall in the guest space. Oh OK, let me use
>> > this value real value which correspond the guests real value."
>>
>> It will take some time for me to understand every thing you have written :)
>
> Ok. Ping me if you have some questions. Reading this might
> help (or confuse you more): http://wiki.xensource.com/xenwiki/XenPVOPSDRM
>
>>
>> >
>> > But the IOMMU has a couple of knobs. One of those is the passthrough
>> > code and the GFX mapping code. The first should be easy to spot (should
>> > tell you on bootup whether you got that enabled or not), while the other
>> > looks to be a bunch of workarounds when passing in GFX cards b/c they ..
>> > well, not sure what, but they look to not work correctly with the IOMMU.
>>
>> By GFX Mapping,
>> Do you mean the code that maps VGA frame-buffers to Guest OS?
>
> No. It is the IOMMU kernel code that checks to see if this is a GFX
> card and if so does something fancy. But the workarounds for this appear
> to be exclusive to Intel chipset so you shouldn't hit any.
>
>>
>> One more silly question on the frame-buffer mapping.
>>
>> I was reading a paper on xen vga pass-though. It says "When applying
>> PCI pass-through, certain memory areas of the physical machine are
>> mapped to the VM. When the guest OS writes to one of those memory
>> addresses, Xen will make sure it is actually written at the
>> appropriate address. This implicates that no other VM is able to make
>> use of that device. The frame-buffer is mapped to the machine's main
>> memory at address 0xA0000 to 0xC0000. This memory range must be mapped
>> to the VM's memory in order for the OS to address the video adapter."
> <nods>
>>
>> My question is....
>> In my case, the host machine is using Nvidia graphics card, so the
>> host frame-buffer memory 0xA0000 to 0xC0000 is being actively used by
>> host. Now if the ATI card maps it's framebuffer to this same address
>> wouldn't it cause any problem.
>
> Ah, but it does not. The first card that is called by the BIOS (so your
> motherboard BIOS) sets itself up to use that memory. The other card has
> to wait. Keep in mind that the A0000 to C0000 is the old style VGA
> framebuffer (80x25) or if you program the card properly it can do
> VESA graphics, but nothing else. Usually the guest gets a Cirrus Logic
> video card passed in which sets this device up (and touching that
> area in the guest ends up in VNC window or SDL window).
>
> Anyhow back to your host..
> If you grep in your kernel log you should see something like:
> [drm] fb mappable at 0xE0142000
>
> This is the framebuffer address.
>>
>> If the the machine has only 1 graphics card those statement make
>> sense, but I could not understand what will happen when there are two
>> graphics cards, each connected to different monitor and assigned to
>> different VM, where would each machines frame-buffer reside in the
>> host memory.
>
> Check the kernel log. Usually it is also the first BAR in the pci device.
>
>>
>> Does Xen allow passing-though multiple graphics cards, each to different VM?
>
> Yes (with some patches posted by the AMD and Intel folks)..
> (look for threads that have some GFX or Radeon or something like that in its title).
>>
>> >
>> > If you see 'identity map for device' where the device is your GFX
>> > then that looks to be the bug. It shouldn't set the identity mapping on it.
>> >
>>
>> I guess the identity map is used only on the Intel chipset. Here are
>> msgs on host machine when addition debugging parameter
>> (amd_iommu_dump) was added.
>>
>> prasad at prasad-kvm:~$ dmesg | grep -i -e iommu -e dmar -e amd
>> [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+
>> root=/dev/sda1 ro iommu=1 amd_iommu_dump
>> [ 0.000000] ACPI: SRAT 00000000bfe9f8b0 00108 (v01 AMD FAM_F_10
>> 00000002 AMD 00000001)
>> [ 0.000000] ACPI: IVRS 00000000bfe9fa00 000D8 (v01 AMD RD890S
>> 00202031 AMD 00000000)
>> [ 0.000000] ACPI: SSDT 00000000bfe9fae0 00DA4 (v01 A M I POWERNOW
>> 00000001 AMD 00000001)
>> [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+
>> root=/dev/sda1 ro iommu=1 amd_iommu_dump
>> [ 0.000000] Please enable the IOMMU option in the BIOS setup
>> [ 0.012918] Performance Events: AMD PMU driver.
>> [ 0.137960] CPU0: AMD Phenom(tm) II X6 1090T Processor stepping 00
>> [ 0.300011] System has AMD C1E enabled
>> [ 2.680386] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300
>> [ 2.680441] AMD-Vi: mmio-addr: 00000000f6000000
>> [ 2.680670] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00
>> [ 2.680726] AMD-Vi: DEV_RANGE_END devid: 00:00.2
>> [ 2.680776] AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00
>> [ 2.680827] AMD-Vi: DEV_SELECT devid: 07:00.0 flags: 00
>> [ 2.680883] AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00
>> [ 2.680934] AMD-Vi: DEV_SELECT devid: 06:00.0 flags: 00
>> [ 2.680984] AMD-Vi: DEV_SELECT devid: 00:05.0 flags: 00
>> [ 2.681035] AMD-Vi: DEV_SELECT devid: 05:00.0 flags: 00
>> [ 2.681086] AMD-Vi: DEV_SELECT devid: 00:06.0 flags: 00
>> [ 2.681137] AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00
>> [ 2.681187] AMD-Vi: DEV_SELECT devid: 00:07.0 flags: 00
>> [ 2.681238] AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00
>> [ 2.681289] AMD-Vi: DEV_SELECT devid: 00:0b.0 flags: 00
>> [ 2.681340] AMD-Vi: DEV_SELECT_RANGE_START devid: 02:00.0 flags: 00
>> [ 2.681391] AMD-Vi: DEV_RANGE_END devid: 02:00.1
>> [ 2.681441] AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00
>> [ 2.681492] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00
>> [ 2.681543] AMD-Vi: DEV_RANGE_END devid: 00:12.2
>> [ 2.681594] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00
>> [ 2.681645] AMD-Vi: DEV_RANGE_END devid: 00:13.2
>> [ 2.681695] AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7
>> [ 2.681746] AMD-Vi: DEV_SELECT devid: 00:14.1 flags: 00
>> [ 2.681797] AMD-Vi: DEV_SELECT devid: 00:14.2 flags: 00
>> [ 2.681847] AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00
>> [ 2.681898] AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00
>> [ 2.681949] AMD-Vi: DEV_ALIAS_RANGE devid: 01:00.0 flags: 00
>> devid_to: 00:14.4
>> [ 2.682010] AMD-Vi: DEV_RANGE_END devid: 01:1f.7
>> [ 2.682855] AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00
>> [ 2.682906] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00
>> [ 2.682957] AMD-Vi: DEV_RANGE_END devid: 00:16.2
>> [ 2.683234] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
>> [ 2.692043] AMD-Vi: Lazy IO/TLB flushing enabled
>> [ 4.360112] ehci_hcd 0000:00:12.2: applying AMD
>> SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
>> [ 4.380635] ehci_hcd 0000:00:13.2: applying AMD
>> SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
>> [ 4.400658] ehci_hcd 0000:00:16.2: applying AMD
>> SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
>> [ 4.811320] powernow-k8: Found 1 AMD Phenom(tm) II X6 1090T
>> Processor (6 cpu cores) (version 2.20.00)
>> [ 12.349919] EDAC amd64_edac: Ver: 3.3.0 Dec 21 2010
>> [ 12.350018] EDAC amd64: This node reports that Memory ECC is
>> currently disabled, set F3x44[22] (0000:00:18.3).
>> [ 12.350030] EDAC amd64: ECC disabled in the BIOS or no ECC
>> capability, module will not load.
>> [ 12.350044] amd64_edac: probe of 0000:00:18.2 failed with error -22
>
> What motherboard is this?
ASUS M4A89TD Pro/USB3, it is mentioned on the link
http://wiki.xensource.com/xenwiki/VTdHowTo
>>
>> prasad at prasad-kvm:~$ lspci | grep -i RV
>> 02:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B64
>> [FireGL V3100 (PCIE)] (rev 80)
>> 02:00.1 Display controller: ATI Technologies Inc RV370 5B64 [FireGL
>> V3100 (PCIE)] (Secondary) (rev 80)
>
> So, how come you aren't passing in 02:00.1?
I tried passing through 02.00.1 as well. When I pass it to a Linux
machine I do not see the device bind to a driver, I don't see the drm
logs in the dmsg. So I thought I am not supposed to pass the 02:00:1.
Should I be passing 02.00.1 instead of 02.00.0?
>>
>> > Look for 'DMAR GFX' on Google. Might shed some more light.
>> >
>>
>> No DMAR for AMD. I think I should have a look at DMAR GFX.
>>
>> I am really thankful to you for your support.
>
> Hopefully I haven't confused you any further :-)
>>
>> Thanks and Regards,
>> Prasad
>
More information about the dri-devel
mailing list