[Intel-gfx] [PATCH v3 00/12] Introduce new methods for verifying ownership in vfio PCI hot reset

[Duan, Zhenzhong]

> -----Original Message-----
> From: Liu, Yi L <mailto:yi.l.liu at intel.com>
> Sent: Saturday, April 1, 2023 10:44 PM
> Subject: [PATCH v3 00/12] Introduce new methods for verifying 
> ownership in vfio PCI hot reset
> 
> VFIO_DEVICE_PCI_HOT_RESET requires user to pass an array of group fds 
> to prove that it owns all devices affected by resetting the calling 
> device. This series introduces several extensions to allow the 
> ownership check better aligned with iommufd and coming vfio device cdev support.
> 
> First, resetting an unopened device is always safe given nobody is 
> using it. So relax the check to allow such devices not covered by 
> group fd array. [1]
> 
> When iommufd is used we can simply verify that all affected devices 
> are bound to a same iommufd then no need for the user to provide extra 
> fd information. This is enabled by the user passing a zero-length fd 
> array and moving forward this should be the preferred way for hot 
> reset. [2]
> 
> However the iommufd method has difficulty working with noiommu devices 
> since those devices don't have a valid iommufd, unless the noiommu 
> device is in a singleton dev_set hence no ownership check is required. 
> [3]
> 
> For noiommu backward compatibility a 3rd method is introduced by 
> allowing the user to pass an array of device fds to prove ownership. 
> [4]
> 
> As suggested by Jason [5], we have this series to introduce the above 
> stuffs to the vfio PCI hot reset. Per the dicussion in [6] [7], in the 
> end of this series, the VFIO_DEVICE_GET_PCI_HOT_RESET_INFO is extended 
> to report devid for the devices opened as cdev. This is goging to 
> support the device fd passing usage.
> 
> The new hot reset method and updated _INFO ioctl are tested with two 
> test commits in below qemu:
> 
> https://github.com/yiliu1765/qemu/commits/iommufd_rfcv3
> (requires to test with cdev kernel)

Test pass with qemu supporting new uAPI, played trick to bypass FLR and trigger
hot reset intentionally. 81:00.0 and 81:00.1 are dependent devices from same slot
and passed to one VM, see below qemu trace for details.

vfio_pci_hot_reset  (0000:81:00.1) multi
vfio_intx_disable_kvm  (0000:81:00.1) KVM INTx accel disabled
vfio_region_mmaps_set_enabled Region 0000:81:00.1 BAR 0 mmaps enabled: 1
vfio_region_mmaps_set_enabled Region 0000:81:00.1 BAR 3 mmaps enabled: 1
vfio_intx_disable  (0000:81:00.1)
vfio_pci_read_config  (0000:81:00.1, @0x44, len=0x2) 0x2008
vfio_pci_read_config  (0000:81:00.1, @0x4, len=0x2) 0x142
vfio_pci_write_config  (0000:81:00.1, @0x4, 0x140, len=0x2)
vfio_pci_hot_reset_has_dep_devices 0000:81:00.1: hot reset dependent devices:
vfio_pci_hot_reset_dep_devices_iommufd  0000:81:00.0 devid 2
vfio_intx_disable_kvm  (0000:81:00.0) KVM INTx accel disabled
vfio_region_mmaps_set_enabled Region 0000:81:00.0 BAR 0 mmaps enabled: 1
vfio_region_mmaps_set_enabled Region 0000:81:00.0 BAR 3 mmaps enabled: 1
vfio_intx_disable  (0000:81:00.0)
vfio_pci_read_config  (0000:81:00.0, @0x44, len=0x2) 0x2008
vfio_pci_read_config  (0000:81:00.0, @0x4, len=0x2) 0x142
vfio_pci_write_config  (0000:81:00.0, @0x4, 0x140, len=0x2)
vfio_pci_hot_reset_dep_devices_iommufd  0000:81:00.1 devid 5
vfio_pci_hot_reset_result 0000:81:00.1 hot reset: Success
vfio_pci_read_config  (0000:81:00.0, @0x3d, len=0x1) 0x1
vfio_intx_enable_kvm  (0000:81:00.0) KVM INTx accel enabled
vfio_intx_enable  (0000:81:00.0)
vfio_pci_read_config  (0000:81:00.1, @0x3d, len=0x1) 0x1
vfio_intx_enable_kvm  (0000:81:00.1) KVM INTx accel enabled
vfio_intx_enable  (0000:81:00.1)
vfio_pci_hot_reset  (0000:81:00.0) multi
vfio_intx_disable_kvm  (0000:81:00.0) KVM INTx accel disabled
vfio_region_mmaps_set_enabled Region 0000:81:00.0 BAR 0 mmaps enabled: 1
vfio_region_mmaps_set_enabled Region 0000:81:00.0 BAR 3 mmaps enabled: 1
vfio_intx_disable  (0000:81:00.0)
vfio_pci_read_config  (0000:81:00.0, @0x44, len=0x2) 0x2008
vfio_pci_read_config  (0000:81:00.0, @0x4, len=0x2) 0x140
vfio_pci_write_config  (0000:81:00.0, @0x4, 0x140, len=0x2)
vfio_pci_hot_reset_has_dep_devices 0000:81:00.0: hot reset dependent devices:
vfio_pci_hot_reset_dep_devices_iommufd  0000:81:00.0 devid 2
vfio_pci_hot_reset_dep_devices_iommufd  0000:81:00.1 devid 5
vfio_intx_disable_kvm  (0000:81:00.1) KVM INTx accel disabled
vfio_region_mmaps_set_enabled Region 0000:81:00.1 BAR 0 mmaps enabled: 1
vfio_region_mmaps_set_enabled Region 0000:81:00.1 BAR 3 mmaps enabled: 1
vfio_intx_disable  (0000:81:00.1)
vfio_pci_read_config  (0000:81:00.1, @0x44, len=0x2) 0x2008
vfio_pci_read_config  (0000:81:00.1, @0x4, len=0x2) 0x140
vfio_pci_write_config  (0000:81:00.1, @0x4, 0x140, len=0x2)
vfio_pci_hot_reset_result 0000:81:00.0 hot reset: Success
vfio_pci_read_config  (0000:81:00.1, @0x3d, len=0x1) 0x1
vfio_intx_enable_kvm  (0000:81:00.1) KVM INTx accel enabled
vfio_intx_enable  (0000:81:00.1)
vfio_pci_read_config  (0000:81:00.0, @0x3d, len=0x1) 0x1
vfio_intx_enable_kvm  (0000:81:00.0) KVM INTx accel enabled
vfio_intx_enable  (0000:81:00.0)
......

vfio_pci_reset  (0000:81:00.1)
vfio_intx_disable_kvm  (0000:81:00.1) KVM INTx accel disabled
vfio_region_mmaps_set_enabled Region 0000:81:00.1 BAR 0 mmaps enabled: 1
vfio_region_mmaps_set_enabled Region 0000:81:00.1 BAR 3 mmaps enabled: 1
vfio_intx_disable  (0000:81:00.1)
vfio_pci_read_config  (0000:81:00.1, @0x44, len=0x2) 0x2008
vfio_pci_read_config  (0000:81:00.1, @0x4, len=0x2) 0x140
vfio_pci_write_config  (0000:81:00.1, @0x4, 0x140, len=0x2)
vfio_pci_hot_reset  (0000:81:00.1) one
vfio_pci_hot_reset_has_dep_devices 0000:81:00.1: hot reset dependent devices:
vfio_pci_hot_reset_dep_devices_iommufd  0000:81:00.0 devid 2
vfio_pci_read_config  (0000:81:00.1, @0x3d, len=0x1) 0x1
vfio_intx_enable_kvm  (0000:81:00.1) KVM INTx accel enabled
vfio_intx_enable  (0000:81:00.1)
vfio_pci_reset  (0000:81:00.0)
vfio_intx_disable_kvm  (0000:81:00.0) KVM INTx accel disabled
vfio_region_mmaps_set_enabled Region 0000:81:00.0 BAR 0 mmaps enabled: 1
vfio_region_mmaps_set_enabled Region 0000:81:00.0 BAR 3 mmaps enabled: 1
vfio_intx_disable  (0000:81:00.0)
vfio_pci_read_config  (0000:81:00.0, @0x44, len=0x2) 0x2008
vfio_pci_read_config  (0000:81:00.0, @0x4, len=0x2) 0x140
vfio_pci_write_config  (0000:81:00.0, @0x4, 0x140, len=0x2)
vfio_pci_hot_reset  (0000:81:00.0) one
vfio_pci_hot_reset_has_dep_devices 0000:81:00.0: hot reset dependent devices:
vfio_pci_hot_reset_dep_devices_iommufd  0000:81:00.0 devid 2
vfio_pci_hot_reset_dep_devices_iommufd  0000:81:00.1 devid 5
vfio_pci_read_config  (0000:81:00.0, @0x3d, len=0x1) 0x1
vfio_intx_enable_kvm  (0000:81:00.0) KVM INTx accel enabled
vfio_intx_enable  (0000:81:00.0)