[PATCH v9 4/5] x86/PCI: Enable a 64bit BAR on AMD Family 15h (Models 30h-3fh) Processors v5

[Boris Ostrovsky]

On 11/23/2017 03:11 AM, Christian König wrote:
> Am 22.11.2017 um 18:27 schrieb Boris Ostrovsky:
>> On 11/22/2017 11:54 AM, Christian König wrote:
>>> Am 22.11.2017 um 17:24 schrieb Boris Ostrovsky:
>>>> On 11/22/2017 05:09 AM, Christian König wrote:
>>>>> Am 21.11.2017 um 23:26 schrieb Boris Ostrovsky:
>>>>>> On 11/21/2017 08:34 AM, Christian König wrote:
>>>>>>> Hi Boris,
>>>>>>>
>>>>>>> attached are two patches.
>>>>>>>
>>>>>>> The first one is a trivial fix for the infinite loop issue, it now
>>>>>>> correctly aborts the fixup when it can't find address space for the
>>>>>>> root window.
>>>>>>>
>>>>>>> The second is a workaround for your board. It simply checks if there
>>>>>>> is exactly one Processor Function to apply this fix on.
>>>>>>>
>>>>>>> Both are based on linus current master branch. Please test if they
>>>>>>> fix
>>>>>>> your issue.
>>>>>> Yes, they do fix it but that's because the feature is disabled.
>>>>>>
>>>>>> Do you know what the actual problem was (on Xen)?
>>>>> I still haven't understood what you actually did with Xen.
>>>>>
>>>>> When you used PCI pass through with those devices then you have made a
>>>>> major configuration error.
>>>>>
>>>>> When the problem happened on dom0 then the explanation is most likely
>>>>> that some PCI device ended up in the configured space, but the routing
>>>>> was only setup correctly on one CPU socket.
>>>> The problem is that dom0 can be (and was in my case() booted with less
>>>> than full physical memory and so the "rest" of the host memory is not
>>>> necessarily reflected in iomem. Your patch then tried to configure that
>>>> memory for MMIO and the system hang.
>>>>
>>>> And so my guess is that this patch will break dom0 on a single-socket
>>>> system as well.
>>> Oh, thanks!
>>>
>>> I've thought about that possibility before, but wasn't able to find a
>>> system which actually does that.
>>>
>>> May I ask why the rest of the memory isn't reported to the OS?
>> That memory doesn't belong to the OS (dom0), it is owned by the 
>> hypervisor.
>>
>>> Sounds like I can't trust Linux resource management and probably need
>>> to read the DRAM config to figure things out after all.
>>
>> My question is whether what you are trying to do should ever be done for
>> a guest at all (any guest, not necessarily Xen).
> 
> The issue is probably that I don't know enough about Xen: What exactly 
> is dom0? My understanding was that dom0 is the hypervisor, but that 
> seems to be incorrect.
> 
> The issue is that under no circumstances *EVER* a virtualized guest 
> should have access to the PCI devices marked as "Processor Function" on 
> AMD platforms. Otherwise it is trivial to break out of the virtualization.
> 
> When dom0 is something like the system domain with all hardware access 
> then the approach seems legitimate, but then the hypervisor should 
> report the stolen memory to the OS using the e820 table.
> 
> When the hypervisor doesn't do that and the Linux kernel isn't aware 
> that there is memory at a given location mapping PCI space there will 
> obviously crash the hypervisor.
> 
> Possible solutions as far as I can see are either disabling this feature 
> when we detect that we are a Xen dom0, scanning the DRAM settings to 
> update Linux resource handling or fixing Xen to report stolen memory to 
> the dom0 OS as reserved.
> 
> Opinions?

You are right, these functions are not exposed to a regular guest.

I think for dom0 (which is a special Xen guest, with additional 
privileges) we may be able to add a reserved e820 region for host memory 
that is not assigned to dom0. Let me try it on Monday (I am out until then).

-boris