[Intel-gfx] IOMMU RMRR for Intel graphic device

[Daniel Vetter]

On Fri, May 10, 2019 at 01:21:59PM +0800, Lu Baolu wrote:
> Forward to i915 mailing list and post the question again so people knows
> what are the key concerns.
> 
> The background is that Linux community is going to collect and report
> the reserved memory ranges of an assigned device to VFIO driver, and any
> mapped address will be checked against the reserved ranges. If there's
> any conflict, vifo will refuse the map request.
> 
> Unfortunately, when it comes Intel graphic device, the conflict happens.
> That means address range mapped through vfio conflicts with the rmrr for
> graphic device.
> 
> https://lkml.org/lkml/2018/6/5/760
> 
> The questions are 1) will the rmrr for graphic device still needs to be
> reserved for BIOS or firmware use, when a device is going to be assigned
> to user level? 2) if no, what's the check point, after which the rmrr is
> unnecessary anymore?

The gfx RMRR isn't for the bios, it's for the driver. It covers stolen
memory, and we need it. There's various piles of hacks to disable use of
stolen, but they're all somewhat fragile, and afaiui for huc/guc we need
it, and huc is part of the uapi exposed by the driver/device combo on
gen9+.

Until our hw folks come up with a better design I think we're just stuck
on this, iow you can't pass-thru modern intel igfx because of this.
-Daniel

> 
> Best regards,
> Lu Baolu
> 
> On 5/9/19 5:14 PM, Zhang, Tina wrote:
> > Hi Baolu,
> > 
> > +Xiong
> > 
> > I think the root cause is that guest i915 needs to access RMRR. Xiong cooked a patch to disable the RMRR use in i915 guest, however that patch didn't get landed into the i915 upstream repo due to some concerns from i915 maintainers. Xiong can give us more backgrounds.
> > 
> > So agreed, need to ask the i915 folk for this.
> > 
> > BR,
> > Tina
> > 
> > 
> > > -----Original Message-----
> > > From: Yuan, Hang
> > > Sent: Thursday, May 9, 2019 4:07 PM
> > > To: Lu Baolu <baolu.lu at linux.intel.com>; Tian, Kevin <kevin.tian at intel.com>;
> > > Zhenyu Wang <zhenyuw at linux.intel.com>; Zhang, Tina
> > > <tina.zhang at intel.com>; Lu, Baolu <baolu.lu at intel.com>; Liu, Yi L
> > > <yi.l.liu at intel.com>
> > > Subject: RE: IOMMU RMRR for Intel graphic device
> > > 
> > > Hi Baolu, as Kevin suggested, would you like to ask i915 people in their
> > > mailing list intel-gfx at lists.freedesktop.org?
> > > 
> > > Regards,
> > > Henry
> > > 
> > > > -----Original Message-----
> > > > From: Lu Baolu [mailto:baolu.lu at linux.intel.com]
> > > > Sent: Thursday, May 9, 2019 2:42 PM
> > > > To: Tian, Kevin <kevin.tian at intel.com>; Zhenyu Wang
> > > > <zhenyuw at linux.intel.com>; Zhang, Tina <tina.zhang at intel.com>; Yuan,
> > > > Hang <hang.yuan at intel.com>; Lu, Baolu <baolu.lu at intel.com>; Liu, Yi L
> > > > <yi.l.liu at intel.com>
> > > > Cc: baolu.lu at linux.intel.com
> > > > Subject: Re: IOMMU RMRR for Intel graphic device
> > > > 
> > > > Hi,
> > > > 
> > > > +Tina and Henry and cc more people
> > > > 
> > > > The background is that Linux community is going to collect and report
> > > > the reserved memory ranges of an assigned device to VFIO driver, and
> > > > any mapped address will be checked against the reserved ranges. If
> > > > there's any conflict, vifo will refuse the map request.
> > > > 
> > > > Unfortunately, when it comes Intel graphic device, the conflict happens.
> > > > That means address range mapped through vfio conflicts with the rmrr
> > > > for graphic device.
> > > > 
> > > > https://lkml.org/lkml/2018/6/5/760
> > > > 
> > > > The questions are 1) will the rmrr for graphic device still needs to
> > > > be reserved for BIOS or firmware use, when a device is going to be
> > > > assigned to user level? 2) if no, what's the check point, after which
> > > > the rmrr is unnecessary anymore?
> > > > 
> > > > Best regards,
> > > > Lu Baolu
> > > > 
> > > > On 5/6/19 2:16 PM, Tian, Kevin wrote:
> > > > > this should better be asked to i915 guys, since it's not
> > > > > virtualization related. :-)
> > > > > 
> > > > > One caveat, iirc, i915 driver tries to reuse stolen memory (covered
> > > > > by
> > > > > RMRR) even after boot time. take it as if another type of memory
> > > > > resource. If true I'm afraid this might be a gap to your proposal.
> > > > > 
> > > > > Since nothing confidential, possibly you can directly discuss in
> > > community.
> > > > > 
> > > > > > From: Lu Baolu [mailto:baolu.lu at linux.intel.com]
> > > > > > Sent: Thursday, May 2, 2019 2:45 PM
> > > > > > 
> > > > > > Ping...
> > > > > > 
> > > > > > Any comments? This has been postponed in the community for a long
> > > > time.
> > > > > > We need to response this as soon as possible.
> > > > > > 
> > > > > > Best regards,
> > > > > > Lu Baolu
> > > > > > 
> > > > > > On 4/29/19 1:19 PM, Lu Baolu wrote:
> > > > > > > Hi Zhenyu,
> > > > > > > 
> > > > > > > As we discussed, BIOS always exports IOMMU reserved memory
> > > regions
> > > > > > > for (a.k.a. RMRR in vt-d spec) Intel integrated graphic device.
> > > > > > > This caused some problems when we pass-through such graphic
> > > > > > > devices to
> > > > user level.
> > > > > > > 
> > > > > > > I am about to propose something to the community so that a RMRR
> > > > > > > for graphic devices could be explicitly canceled as long as the
> > > > > > > driver
> > > > > > > (i915 or vfio) knows that the RMRR will never be used by BIOS
> > > anymore.
> > > > > > > 
> > > > > > > The same story happens for USB host controller devices. And since
> > > > > > > we know that BIOS will stop using that memory region as soon as
> > > > > > > the driver clears the SMI bits.
> > > > > > > 
> > > > > > > So the question is, can graphic driver know when the RMRR for
> > > > > > > graphic could be canceled?
> > > > > > > 
> > > > > > > Best regards,
> > > > > > > Lu Baolu
> > > > > > > 
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch