[Intel-gfx] About the iGVT-g's requirement to pin guest contexts in VM

[Wang, Zhi A]

Attach the big picture to help the discussion:

Windows Guest     Linux Guest      Linux Guest (i915 guest mode)  * We are here
+--------------+ +--------------+ +-------------------------------------------+ 
|              | |              | |    Guest Context Lifecycle Management     |
|Windows Driver| |     i915     | | +---------------------------------------+ |
|              | |          +---->| |        CREATE/DESTROY/PIN/UNPIN       | |
| (GUEST MODE) | | (GUEST MODE) | | | +----------------+ +----------------+ | | 
|              | |              | | | |Execlist Context| |Execlist Context| | | 
|              | |              | | | +-----^----------+ +------^---------+ | |
|              | |              | | +-------|-------------------|-----------+ | 
+--------------+ +--------------+ +---------|-------------------|-------------+
                                            |    NOTIFICATION   | 
                  +-------------------------|-------------------|-------------+
                  |               XenGT Shadow Context Lifecycle|Management   |
                  |                         |                   |             | 
                  |               +---------|-------------------|----------+  |  
                  | +-----------+ | +-------v-------+ +---------v------+   |  | 
                  | | i915      | | | Shadow Context| | Shadow Context |   |  | 
                  | | Host Mode | | +---------------+ +----------------+   |  |
                  | +-----------+ +----------------------------------------+  | 
                  |             DOM0 Linux (i915 host mode w/XenGT)           | 
                  +-----------------------------------------------------------+
                  +-----------------------------------------------------------+ 
                  |                     Hypervisor                            | 
                  +-----------------------------------------------------------+

SHADOW CONTEXT SUBMISSION

As you can see, in this picture, each guest execlist context will have an related shadow context in host, and XenGT will be responsible for
a. update SHADOW context via GUEST context when guest wants to submit an workload.
b. submitting the shadow context into hardware.
c. update the guest context via shadow context, when shadow context is finished.
d. inject virtual context switch interrupt into guest.

Then guest will see "hey, my job was retired from hardware, then I can do something to my context."

NOTIFICATION BETWEEN HOST AND GUEST

Now in our design we have built a notification channel for guest to notify XenGT due to performance reason.
With this channel, guest can play like this "Hey XenGT, I created a new context, please shadow it! Oh, I have destroyed the context, please stop tracking this context."

But when this trick comes before guest pin/unpin, it has problems.

PROBLEMS

First, guest context pin/unpin will cause LRCA change, which breaks relationship between guest context and shadow context.
As you can see that in our design, XenGT needs an approach to find the related shadow context according to something of an guest context.
For now we find the related shadow context via guest context ID(LRCA in fact).
But whatever it is, I think there should be something unique.

XenGT has no knowledge about guest context pin/unpin. Guest may swap out an context if it sees the seqno has been passed.
When the XenGT wants to access guest context, the guest context may be not there (swapped-out already).

It's hard and complicated for XenGT to track guest context without an unique context ID and a stable execlist context backing store.
For now the whole tricks are only works under virtualization environment and will not affect the native i915. 

Welcome to discussions! :)

Thanks,
Zhi.

-----Original Message-----
From: Chris Wilson [mailto:chris at chris-wilson.co.uk] 
Sent: Monday, August 24, 2015 6:23 PM
To: intel-gfx at lists.freedesktop.org; igvt-g at lists.01.org; Wang, Zhi A; Tian, Kevin; joonas.lahtinen at linux.intel.com
Subject: Re: About the iGVT-g's requirement to pin guest contexts in VM

On Mon, Aug 24, 2015 at 06:04:28PM +0800, Zhiyuan Lv wrote:
> Hi Chris,
> 
> On Thu, Aug 20, 2015 at 09:36:00AM +0100, Chris Wilson wrote:
> > On Thu, Aug 20, 2015 at 03:45:21PM +0800, Zhiyuan Lv wrote:
> > > Intel GVT-g will perform EXECLIST context shadowing and ring 
> > > buffer shadowing. The shadow copy is created when guest creates a context.
> > > If a context changes its LRCA address, the hypervisor is hard to 
> > > know whether it is a new context or not. We always pin context 
> > > objects to global GTT to make life easier.
> > 
> > Nak. Please explain why we need to workaround a bug in the host. We 
> > cannot pin the context as that breaks userspace (e.g. synmark) who 
> > can and will try to use more contexts than we have room.
> 
> Could you have a look at below reasons and kindly give us your inputs?
> 
> 1, Due to the GGTT partitioning, the global graphics memory available 
> inside virtual machines is much smaller than native case. We cannot 
> support some graphics memory intensive workloads anyway. So it looks 
> affordable to just pin contexts which do not take much GGTT.

Wrong. It exposes the guest to a trivial denial-of-service attack. A smaller GGTT does not actually limit clients (there is greater aperture pressure and some paths are less likely but an individual client will function just fine).
 
> 2, Our hypervisor needs to change i915 guest context in the shadow 
> context implementation. That part will be tricky if the context is not 
> always pinned. One scenario is that when a context finishes running, 
> we need to copy shadow context, which has been updated by hardware, to 
> guest context. The hypervisor knows context finishing by context 
> interrupt, but that time shrinker may have unpin the context and its 
> backing storage may have been swap-out. Such copy may fail.

That is just a bug in your code. Firstly allowing swapout on an object you still are using, secondly not being able to swapin.
-Chris

--
Chris Wilson, Intel Open Source Technology Centre