[RFC PATCH v2] Utilize the PCI API in the TTM framework.

[Konrad Rzeszutek Wilk]

. snip ..
> >>>I think the error path would be the same in both cases?
> >>Not really. The really dangerous situation is if TTM is allowed to
> >>exhaust all GFP_KERNEL memory. Then any application or kernel task
> >Ok, since GFP_KERNEL does not contain the GFP_DMA32 flag then
> >this should be OK?
> 
> No, Unless I miss something, on a machine with 4GB or less,
> GFP_DMA32 and GFP_KERNEL are allocated from the same pool of pages?

Yes. Depending on the E820 and where the PCI hole is present. More
details below.
> 
> >
> >>What *might* be possible, however, is that the GFP_KERNEL memory on
> >>the host gets exhausted due to extensive TTM allocations in the
> >>guest, but I guess that's a problem for XEN to resolve, not TTM.
> >Hmm. I think I am missing something here. The GFP_KERNEL is any memory
> >and the GFP_DMA32 is memory from the ZONE_DMA32. When we do start
> >using the PCI-API, what happens underneath (so under Linux) is that
> >"real PFNs" (Machine Frame Numbers) which are above the 0x100000 mark
> >get swizzled in for the guest's PFNs (this is for the PCI devices
> >that have the dma_mask set to 32bit). However, that is a Xen MMU
> >accounting issue.
> 
> 
> So I was under the impression that when you allocate coherent memory
> in the guest, the physical page comes from DMA32 memory in the host.

No. It comes from DMA32 zone off the hypervisor pool. If say you have a machine
with 24GB, the first guest (Dom0) could allocate memory from 20->24GB
(so only 4GB allocated to it). It will then also fetch 64MB from the DMA32
zone for the SWIOTLB. Then the next guest, say 4GB (gets 16GB->20GB) - gets
64MB from DMA32. And  so on.

So at the end we have 16GB taken from 8GB->24GB, and 320MB taken from 
0->4GB. When you start allocating coherent memory from each guest
(and yeah, say we use 2GB each), we end up with the first guest getting
the 2GB, the second getting 1.7GB, and then the next two getting zil.

You still have GFP_KERNEL memory in each guest - the first one has 2GB left
, then second 2.3, the next two have each 4GB.