[Intel-gfx] About the iGVT-g's requirement to pin guest contexts in VM
Wang, Zhi A
zhi.a.wang at intel.com
Wed Aug 26 09:42:26 PDT 2015
Attach the some introduction to help the discussion:
[ Current Implementation - Performance Oriented ! ]
Guest create Guest submits
a new context its context
+ + ^
| | |
+-------+ |Notify XenGT | |
| Guest | | | |
+-------+ | | | TIMELINE
==================v====================================v=====+=============>
+-------+ <------------------------------->
| XenGT | + + ^
+-------+ | +-----------------------------+ | |
| | *XENGT TRACK THE WHOLE | | |
Start to track | | LIFE CYCLE OF GUEST CONTEXT!| | |
guest context | +-----------------------------+ | |
| | |
v v +
XenGT creates XenGT submits XenGT updates
Shadow context shadow context guest context
[Track the whole life cycle of guest context via notification channel between guest and host]
ADVANTAGE
*Avoid CPU usage peak at the time of guest context submission due to shadow context translation
- Build shadow context as early as possible
- No need to postpone the shadow context translation into the time of guest context submission
*Avoid the cost of frequently re-recreating the shadow context
- XenGT just create/destroy shadow context with message from notification channel
- No need to re-create / destroy shadow context in each time of guest context submission(PERFORMANCE!)
DISADVANTAGE
* Need an unique identification to locate the shadow context via information from guest context. Current it's LRCA.
- Break by dynamically context GGTT pin/unpin in i915
* Guest needs extra modification of leveraging the notification channel to assist XenGT manage the lifecycle of shadow context.
- Introduce extra functions into guest driver: i915_{execlist, ppgtt}_notify_vgt()
* Backing store of GEM object has to be pinned, as hypervisor tracks guest memory via pfn.
- Break by: shmem(tmpfs) swap-in/out will cause context pfn change when there are no available pages in page cache and swap cache.
SOLUTION
* Pin guest context to get an unique LRCA and stable backing store.
- Windows driver has behave like this already without extra modification.
[ Simple Implantation - Architecture Clean Oriented ! ]
Guest sees updated
Guest creates Guest submits context via virtual
context context context switch interrupt
+ + ^
| | |
+-----------+ | | |
| Guest | | | |
+-----------+ | | | TIMELINE
=================v================v==============+=========================>
+------------+ |
| DOM0 XenGT | + ^ |
+------------+ | | XenGT update |
+-------------+ | guest context |
| | via shadow |
| | context |
| | |
| | |
v + |
XenGT creates XenGT submits XenGT destroy shadow
shadow context +-----> shadow context context
from guest into hardware
context
+-------------------------------+
| * IN-SUBMISSION-TIME |
| SHADOW CONTEXT CREATE/DESTROY |
+-------------------------------+
ADVANTAGE
* No need to use notification channel
- Avoid introduce extra modification into i915
* No need to pin guest GEM object
- Guest backing store is pinned by i915 at the time of guest context submission.
- Shadow context only survive in the time of submission, so unique identification between guest context and shadow context is not needed.
DISADVANTAGE
* Frequently shadow context re-create/destroy at the time of guest context submission(PERFORMANCE! MEMORY FOOTPRINT!)
- Even trivial screen refresh will trigger guest context submission, like moving mouse over an event-sensed button in X-window.
- May cause heavy performance drop and heavy memory footprints, due to frequently shadow context re-create/destroy.
* High CPU usage peak due to frequently shadow context re-create/destroy at the time of guest context submission.
Hope to help the discussion! :)
Thanks,
Zhi.
-----Original Message-----
From: Wang, Zhi A
Sent: Tuesday, August 25, 2015 1:19 AM
To: Chris Wilson; intel-gfx at lists.freedesktop.org; igvt-g at lists.01.org; Tian, Kevin; joonas.lahtinen at linux.intel.com
Subject: RE: About the iGVT-g's requirement to pin guest contexts in VM
Attach the big picture to help the discussion:
Windows Guest Linux Guest Linux Guest (i915 guest mode) * We are here
+--------------+ +--------------+ +-------------------------------------------+
| | | | | Guest Context Lifecycle Management |
|Windows Driver| | i915 | | +---------------------------------------+ |
| | | +---->| | CREATE/DESTROY/PIN/UNPIN | |
| (GUEST MODE) | | (GUEST MODE) | | | +----------------+ +----------------+ | |
| | | | | | |Execlist Context| |Execlist Context| | |
| | | | | | +-----^----------+ +------^---------+ | |
| | | | | +-------|-------------------|-----------+ |
+--------------+ +--------------+ +---------|-------------------|-------------+
| NOTIFICATION |
+-------------------------|-------------------|-------------+
| XenGT Shadow Context Lifecycle|Management |
| | | |
| +---------|-------------------|----------+ |
| +-----------+ | +-------v-------+ +---------v------+ | |
| | i915 | | | Shadow Context| | Shadow Context | | |
| | Host Mode | | +---------------+ +----------------+ | |
| +-----------+ +----------------------------------------+ |
| DOM0 Linux (i915 host mode w/XenGT) |
+-----------------------------------------------------------+
+-----------------------------------------------------------+
| Hypervisor |
+-----------------------------------------------------------+
SHADOW CONTEXT SUBMISSION
As you can see, in this picture, each guest execlist context will have an related shadow context in host, and XenGT will be responsible for
a. update SHADOW context via GUEST context when guest wants to submit an workload.
b. submitting the shadow context into hardware.
c. update the guest context via shadow context, when shadow context is finished.
d. inject virtual context switch interrupt into guest.
Then guest will see "hey, my job was retired from hardware, then I can do something to my context."
NOTIFICATION BETWEEN HOST AND GUEST
Now in our design we have built a notification channel for guest to notify XenGT due to performance reason.
With this channel, guest can play like this "Hey XenGT, I created a new context, please shadow it! Oh, I have destroyed the context, please stop tracking this context."
But when this trick comes before guest pin/unpin, it has problems.
PROBLEMS
First, guest context pin/unpin will cause LRCA change, which breaks relationship between guest context and shadow context.
As you can see that in our design, XenGT needs an approach to find the related shadow context according to something of an guest context.
For now we find the related shadow context via guest context ID(LRCA in fact).
But whatever it is, I think there should be something unique.
XenGT has no knowledge about guest context pin/unpin. Guest may swap out an context if it sees the seqno has been passed.
When the XenGT wants to access guest context, the guest context may be not there (swapped-out already).
It's hard and complicated for XenGT to track guest context without an unique context ID and a stable execlist context backing store.
For now the whole tricks are only works under virtualization environment and will not affect the native i915.
Welcome to discussions! :)
Thanks,
Zhi.
-----Original Message-----
From: Chris Wilson [mailto:chris at chris-wilson.co.uk]
Sent: Monday, August 24, 2015 6:23 PM
To: intel-gfx at lists.freedesktop.org; igvt-g at lists.01.org; Wang, Zhi A; Tian, Kevin; joonas.lahtinen at linux.intel.com
Subject: Re: About the iGVT-g's requirement to pin guest contexts in VM
On Mon, Aug 24, 2015 at 06:04:28PM +0800, Zhiyuan Lv wrote:
> Hi Chris,
>
> On Thu, Aug 20, 2015 at 09:36:00AM +0100, Chris Wilson wrote:
> > On Thu, Aug 20, 2015 at 03:45:21PM +0800, Zhiyuan Lv wrote:
> > > Intel GVT-g will perform EXECLIST context shadowing and ring
> > > buffer shadowing. The shadow copy is created when guest creates a context.
> > > If a context changes its LRCA address, the hypervisor is hard to
> > > know whether it is a new context or not. We always pin context
> > > objects to global GTT to make life easier.
> >
> > Nak. Please explain why we need to workaround a bug in the host. We
> > cannot pin the context as that breaks userspace (e.g. synmark) who
> > can and will try to use more contexts than we have room.
>
> Could you have a look at below reasons and kindly give us your inputs?
>
> 1, Due to the GGTT partitioning, the global graphics memory available
> inside virtual machines is much smaller than native case. We cannot
> support some graphics memory intensive workloads anyway. So it looks
> affordable to just pin contexts which do not take much GGTT.
Wrong. It exposes the guest to a trivial denial-of-service attack. A smaller GGTT does not actually limit clients (there is greater aperture pressure and some paths are less likely but an individual client will function just fine).
> 2, Our hypervisor needs to change i915 guest context in the shadow
> context implementation. That part will be tricky if the context is not
> always pinned. One scenario is that when a context finishes running,
> we need to copy shadow context, which has been updated by hardware, to
> guest context. The hypervisor knows context finishing by context
> interrupt, but that time shrinker may have unpin the context and its
> backing storage may have been swap-out. Such copy may fail.
That is just a bug in your code. Firstly allowing swapout on an object you still are using, secondly not being able to swapin.
-Chris
--
Chris Wilson, Intel Open Source Technology Centre
More information about the Intel-gfx
mailing list