[PATCH v3 0/2] VFIO mdev aggregated resources handling

[Cornelia Huck]

On Fri, 10 Jul 2020 09:58:49 +0800
Yan Zhao <yan.y.zhao at intel.com> wrote:

> On Thu, Jul 09, 2020 at 02:22:26PM -0600, Alex Williamson wrote:
> > On Thu, 9 Jul 2020 15:23:34 +0800
> > Yan Zhao <yan.y.zhao at intel.com> wrote:
> >   
> > > On Thu, Jul 09, 2020 at 02:53:05AM +0000, Tian, Kevin wrote:
> > > 
> > > <...>  
> > > > > We also can't even seem to agree that type is a necessary requirement
> > > > > for compatibility.  Your discussion below of a type-A, which is
> > > > > equivalent to a type-B w/ aggregation set to some value is an example
> > > > > of this.  We might also have physical devices with extensions to
> > > > > support migration.  These could possibly be compatible with full mdev
> > > > > devices.  We have no idea how an administrative tool would discover
> > > > > this other than an exhaustive search across every possible target.
> > > > > That's ugly but feasible when considering a single target host, but
> > > > > completely untenable when considering a datacenter.    
> > > > 
> > > > If exhaustive search can be done just one-off to build the compatibility
> > > > database for all assignable devices on each node, then it might be
> > > > still tenable in datacenter?    
> > > yes, Alex, do you think below behavior to build compatibility database is
> > > acceptable?
> > > 
> > > management stack could do the exhaustive search in one shot to build the
> > > compatibility database for all devices in every node. Meanwhile, it caches
> > > migration version strings for all tested devices.
> > > when there's a newly created/attached device, management stack could write
> > > every cached strings to migration version attribute of the newly
> > > created/attached device in order to update the migration compatibility
> > > database. Then it caches the migration version string of the newly
> > > created/attached device as well.
> > > once a device attribute is modified, e.g. after changing its aggregation
> > > count or updating its parent driver, update its cached migration version
> > > string and update the compatibility database by testing against migration
> > > version attribute of this device.  
> > 
> > This is exactly the scenario that I think is untenable.  You're asking
> > a management application to keep a live database of the opaque version
> > string of every device type and likely every instance of a device,
> > which it's not allowed to compare on its own, it can only pipe them  
> if management stack is allowed to compare on its own, then the migration
> version strings have to be standardized.
> 
> But it's a little hard to do it.
> e.g. 
> for GVT, string format can be
> "parent device PCI ID" + "version of gvt driver" + "mdev type" +
> "aggregator count".
> 
> for an NVMe VF connecting to a remote storage. it is
> "PCI ID" + "driver version" + "configured remote storage URL"
> 
> for a QAT VF, it's
> "PCI ID" + "driver version" + "supported encryption set".
> 
> The management stack also needs to understand how to compare those
> strings, which is also hard. e.g.
> two device with different PCI IDs are incompatible initially, but later
> because of software upgrade, they are compatible again.

One thing that is bothering me here is the amount of information that
is supposed to be available to whomever is checking the compatibility.

This seems to differ a lot between different classes of devices. The
examples you cited assume that a lot of information is available that
can be exposed somewhere. Now, when I look at vfio-ccw, there's hardly
any information available at the subchannel level. We do not know if we
migrate a disk to another matching disk (or the same one), or whether
we're trying to migrate a disk to a tape device. Any management
application trying to migrate vfio-ccw devices has to trust information
provided by the admin, and we only know that something's wrong if the
guest gets confused about the device.

So, I'm wondering if we're not overengineering here. Management
applications would have to support a wide range of information that is
available or not, and would still have to trust the admin for some
cases. It seems hard to fit all of that under the same umbrella.

We also cannot rely on the vendor driver to exhaustively determine
compatibility, as it may not have that information available, either.
We could still try to migrate to an incompatible device because we
simply do not know about that beforehand.