[PATCH v2 05/11] vfio: Use IOMMU_CAP_ENFORCE_CACHE_COHERENCY for vfio_file_enforced_coherent()

[Tian, Kevin]

> From: Jason Gunthorpe <jgg at nvidia.com>
> Sent: Tuesday, November 8, 2022 8:53 AM
> 
> iommufd doesn't establish the iommu_domains until after the device FD is
> opened, even if the container has been set. This design is part of moving
> away from the group centric iommu APIs.
> 
> This is fine, except that the normal sequence of establishing the kvm
> wbinvd won't work:
> 
>    group = open("/dev/vfio/XX")
>    ioctl(group, VFIO_GROUP_SET_CONTAINER)
>    ioctl(kvm, KVM_DEV_VFIO_GROUP_ADD)
>    ioctl(group, VFIO_GROUP_GET_DEVICE_FD)
> 
> As the domains don't start existing until GET_DEVICE_FD. Further,
> GET_DEVICE_FD requires that KVM_DEV_VFIO_GROUP_ADD already be
> done as that
> is what sets the group->kvm and thus device->kvm for the driver to use
> during open.
> 
> Now that we have device centric cap ops and the new
> IOMMU_CAP_ENFORCE_CACHE_COHERENCY we know what the
> iommu_domain will be
> capable of without having to create it. Use this to compute
> vfio_file_enforced_coherent() and resolve the ordering problems.
> 
> VFIO always tries to upgrade domains to enforce cache coherency, it never
> attaches a device that supports enforce cache coherency to a less capable
> domain, so the cap test is a sufficient proxy for the ultimate
> outcome. iommufd also ensures that devices that set the cap will be
> connected to enforcing domains.
> 
> Tested-by: Nicolin Chen <nicolinc at nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg at nvidia.com>

Reviewed-by: Kevin Tian <kevin.tian at intel.com>