[Intel-gfx] About GVT-g Interrupt in i915 Host

[Wang, Zhi A]

BACKGROUND

Under Intel GVT-g environment, HW interrupts are shared among different VMs.

Because of the sharing, to enable or disable a HW interrupt becomes a bit
complicated. Considered that a virtual machine may request to disable a
interrupt which may be using by other virtual machines. The GEN IRQ register
value has to be re-calculated before it gets updated to HW.

i.e. 

- Virtual machine(VM) A wants to *disable* a PIPE VBLANK interrupt.
- Virtual machine B wants to *enable* a PIPE VBLANK interrupt.

The final result is to *enable* the HW VBLANK interrupt, so that we can:
- Get VBLANK interrupt from HW for virtual machine B.

Meanwhile, we choose not to inject VBLANK interrupt into virtual machine A,
which actually doesn't want VBLANK interrupt at this time.

Current Implementation - Interrupt Injection in GVT-g

 +-------------------------------+ +------------------+
 | Linux Guest                   | | Linux Guest      |
 | I915 (Guest Mode) *Upstreamed | | i915 (Guest Mode)|
 +--------------------^----------+ +----^-------------+
                      |                 |
             INJECT!  |                 | INJECT!
                      |                 |
+---------------------|-----------------|------------+
| +------------+  +-------------------------------+  |
| |i915 *      |  |      GVT-g                    |  |
| |(Host Mode) |  +-+-----------------------------+  |
| +-----^------+    |    Host Linux Kernel           |
+-------|-----------|----^---------------------------+
        +-----------+    |
         INJECT!         |  Interrupts
+------------------------|---------------------------+
|                    Hypervisor                      |
+------------------------|---------------------------+
+------------------------|---------------------------+
|                   GPU Hardware                     |
+----------------------------------------------------+

Current implementation treats i915 host(DOM0) like a VM, all the interrupt
register access will be trapped and emulated by GVT-g.

The most obvious advantange is minimum i915 host code change:

- add some hooks in I915_READ()/I915_WRITE() to trap host common register access
- add some hooks in i915 irq initilization path to trap host interrupt
- add some hooks in i915_gem_gtt.c to trap host GTT mmio access.

But it brings some architectural conflict. so we purpose another approach.

EXPECTED ARCHITECTURE

 +-------------------------------+ +------------------+
 | Linux Guest                   | | Linux Guest      |
 | I915 (Guest Mode) *Upstreamed | | i915 (Guest Mode)|
 +--------------------^----------+ +--------^---------+
                      |                     |
             INJECT!  +------------------+  | INJECT!
                                         |  |
+----------------------------------------|--|---------+
| +-------------------------+           +|--|-------+ |
| |i915                     +----------->   GVT-g   | |
| |(Host Mode)              | DISPATCH! +-----------+ |
| +-------------------^-----+                         |
+---------------------|-------------------------------+
                      | Interrupt
+---------------------|-------------------------------+
|                    Hypervisor                       |
+-----------------------------------------------------+
+---------------------|-------------------------------+
|                   GPU Hardware                      |
+-----------------------------------------------------+

             [Host i915 manages all interrupts]

DESIGN

To achieve such kind of an approach, an ideal design in i915 is:

Introduce two IRQ register caches into i915, one for host and one for GVT-g.

Then we provide an central IRQ register mainpulation interface to other
components of host i915, so we can log all interrupt register values of
host i915.

Meanwhile, GVT-g will calculate the IRQ register values according to all
virtual machines and update GVT-g IRQ register cache inside i915 host.

When i915 host itself wants to update IRQ registers or GVT-g requests i915 host
to update IRQ registers, all related IRQ registers of i915 host and GVT-g
will be re-calculated and combined together and written into HW later.

This figure shows the details of the design above

+----------------------------------------------------+                          
|    DRM / i915 components IRQ manipulation clients  |                          
|              drm_irq_{install,uninstall}...        |                          
+--------------------------^-------------------------+                          
                           |                                         +-------+  
+--------------------------|-------------------------+               |       |  
|  i915: i915_irq.c        |                         |            +----Guest |  
|    +---------------------v-----------------------+ |  COMBINE   |  |       |  
|    | Central IRQ register manipulation interface | |  ALL GUEST |  +-------+  
|    +---------------------------------------------+ |   vIRQ     |             
|                                                    | REGISTERS! |             
|    +--------------------+   +--------------------+ | +-------+  |  +-------+   
|    |      i915 host     |   |      GVT-g         | | |       |  |  |       |  
|    | IRQ Register Cache |   | IRQ Register Cache <-----GVT-g <--+----Guest |  
|    |                    |   |                    | | |       |     |       |  
|    +--------+-----------+   +---------+----------+ | +-------+     +-------+  
|             |      COMBINE BOTH!      |            |                          
|             +------------|------------+            |                          
|                          |                         |                          
|             +------------v------------+  *Cache    |                          
|             |  *HW IRQ Register Cache |  combined  |                          
|             +------------+----------- +  values    |                          
+--------------------------|-------------------------+                          
                           |                                                    
+--------------------------v-------------------------+                          
|                      GPU Hardware                  |                          
+----------------------------------------------------+                          
       [ Another approach without trapping i915 host ]

A SIMPLER DESIGN
 
We have been cooked some PoC patches based on above ideal design. After some discussions with Joonas,
we thought it was still a bit complicated. As in the design, we have to record i915 host irq register
values, we have to introduce a framework to support the interrupt register value logging. And the
major disadvantage is it will affect the native code path, mean while LOC is also a problem.
 
So we think over and over again to see if it's possible for GVT-g to require less interrupt from i915
host.
 
As we are planning to do full display virtualization. For guest, all display related interrupts
are emulated. So GVT-g doesn't need i915 host pass it these interrupts. For now, it seems GVT-g only
needs i915 host forwards some GT interrupts used to emulate the status of guest workload execustion:
 
- PIPE_CONTROL_NOTIFY used by Windows guest
- MI_USER_INTERRUPT used by Linux guest
 
Let these two interrupt always be enabled under i915 host mode.
 
- CONTEXT SWITCH interrupt
 
This interrupt has been always enablined under i915 host mode. So no extra modification is needed.

Then call vgt interrupt event forwarding handler at the end of gen8_irq_handler.
 
Maybe this should be simplest design.
 
Welcome to discuss!

Thanks,
Zhi.