[Intel-gfx] About GVT-g Interrupt in i915 Host

Wang, Zhi A zhi.a.wang at intel.com
Wed Sep 9 07:54:34 PDT 2015


Hi Daniel:
    Really appreciate the reply! :) I just read the code again. And we will prepare the PoC patch based on the new ideas! :)

-----Original Message-----
From: Daniel Vetter [mailto:daniel.vetter at ffwll.ch] On Behalf Of Daniel Vetter
Sent: Wednesday, September 09, 2015 4:40 AM
To: Wang, Zhi A
Cc: intel-gfx at lists.freedesktop.org; Vetter, Daniel
Subject: Re: [Intel-gfx] About GVT-g Interrupt in i915 Host

On Tue, Sep 08, 2015 at 04:03:02PM +0000, Wang, Zhi A wrote:
> BACKGROUND
> 
> Under Intel GVT-g environment, HW interrupts are shared among different VMs.
> 
> Because of the sharing, to enable or disable a HW interrupt becomes a bit
> complicated. Considered that a virtual machine may request to disable a
> interrupt which may be using by other virtual machines. The GEN IRQ register
> value has to be re-calculated before it gets updated to HW.
> 
> i.e. 
> 
> - Virtual machine(VM) A wants to *disable* a PIPE VBLANK interrupt.
> - Virtual machine B wants to *enable* a PIPE VBLANK interrupt.
> 
> The final result is to *enable* the HW VBLANK interrupt, so that we can:
> - Get VBLANK interrupt from HW for virtual machine B.
> 
> Meanwhile, we choose not to inject VBLANK interrupt into virtual machine A,
> which actually doesn't want VBLANK interrupt at this time.
> 
> Current Implementation - Interrupt Injection in GVT-g
> 
>  +-------------------------------+ +------------------+
>  | Linux Guest                   | | Linux Guest      |
>  | I915 (Guest Mode) *Upstreamed | | i915 (Guest Mode)|
>  +--------------------^----------+ +----^-------------+
>                       |                 |
>              INJECT!  |                 | INJECT!
>                       |                 |
> +---------------------|-----------------|------------+
> | +------------+  +-------------------------------+  |
> | |i915 *      |  |      GVT-g                    |  |
> | |(Host Mode) |  +-+-----------------------------+  |
> | +-----^------+    |    Host Linux Kernel           |
> +-------|-----------|----^---------------------------+
>         +-----------+    |
>          INJECT!         |  Interrupts
> +------------------------|---------------------------+
> |                    Hypervisor                      |
> +------------------------|---------------------------+
> +------------------------|---------------------------+
> |                   GPU Hardware                     |
> +----------------------------------------------------+
> 
> Current implementation treats i915 host(DOM0) like a VM, all the interrupt
> register access will be trapped and emulated by GVT-g.
> 
> The most obvious advantange is minimum i915 host code change:
> 
> - add some hooks in I915_READ()/I915_WRITE() to trap host common register access
> - add some hooks in i915 irq initilization path to trap host interrupt
> - add some hooks in i915_gem_gtt.c to trap host GTT mmio access.
> 
> But it brings some architectural conflict. so we purpose another approach.
> 
> EXPECTED ARCHITECTURE
> 
>  +-------------------------------+ +------------------+
>  | Linux Guest                   | | Linux Guest      |
>  | I915 (Guest Mode) *Upstreamed | | i915 (Guest Mode)|
>  +--------------------^----------+ +--------^---------+
>                       |                     |
>              INJECT!  +------------------+  | INJECT!
>                                          |  |
> +----------------------------------------|--|---------+
> | +-------------------------+           +|--|-------+ |
> | |i915                     +----------->   GVT-g   | |
> | |(Host Mode)              | DISPATCH! +-----------+ |
> | +-------------------^-----+                         |
> +---------------------|-------------------------------+
>                       | Interrupt
> +---------------------|-------------------------------+
> |                    Hypervisor                       |
> +-----------------------------------------------------+
> +---------------------|-------------------------------+
> |                   GPU Hardware                      |
> +-----------------------------------------------------+
> 
>              [Host i915 manages all interrupts]
> 
> DESIGN
> 
> To achieve such kind of an approach, an ideal design in i915 is:
> 
> Introduce two IRQ register caches into i915, one for host and one for GVT-g.
> 
> Then we provide an central IRQ register mainpulation interface to other
> components of host i915, so we can log all interrupt register values of
> host i915.
> 
> Meanwhile, GVT-g will calculate the IRQ register values according to all
> virtual machines and update GVT-g IRQ register cache inside i915 host.
> 
> When i915 host itself wants to update IRQ registers or GVT-g requests i915 host
> to update IRQ registers, all related IRQ registers of i915 host and GVT-g
> will be re-calculated and combined together and written into HW later.
> 
> This figure shows the details of the design above
> 
> +----------------------------------------------------+                          
> |    DRM / i915 components IRQ manipulation clients  |                          
> |              drm_irq_{install,uninstall}...        |                          
> +--------------------------^-------------------------+                          
>                            |                                         +-------+  
> +--------------------------|-------------------------+               |       |  
> |  i915: i915_irq.c        |                         |            +----Guest |  
> |    +---------------------v-----------------------+ |  COMBINE   |  |       |  
> |    | Central IRQ register manipulation interface | |  ALL GUEST |  +-------+  
> |    +---------------------------------------------+ |   vIRQ     |             
> |                                                    | REGISTERS! |             
> |    +--------------------+   +--------------------+ | +-------+  |  +-------+   
> |    |      i915 host     |   |      GVT-g         | | |       |  |  |       |  
> |    | IRQ Register Cache |   | IRQ Register Cache <-----GVT-g <--+----Guest |  
> |    |                    |   |                    | | |       |     |       |  
> |    +--------+-----------+   +---------+----------+ | +-------+     +-------+  
> |             |      COMBINE BOTH!      |            |                          
> |             +------------|------------+            |                          
> |                          |                         |                          
> |             +------------v------------+  *Cache    |                          
> |             |  *HW IRQ Register Cache |  combined  |                          
> |             +------------+----------- +  values    |                          
> +--------------------------|-------------------------+                          
>                            |                                                    
> +--------------------------v-------------------------+                          
> |                      GPU Hardware                  |                          
> +----------------------------------------------------+                          
>        [ Another approach without trapping i915 host ]
> 
> A SIMPLER DESIGN
>  
> We have been cooked some PoC patches based on above ideal design. After some discussions with Joonas,
> we thought it was still a bit complicated. As in the design, we have to record i915 host irq register
> values, we have to introduce a framework to support the interrupt register value logging. And the
> major disadvantage is it will affect the native code path, mean while LOC is also a problem.
>  
> So we think over and over again to see if it's possible for GVT-g to require less interrupt from i915
> host.
>  
> As we are planning to do full display virtualization. For guest, all display related interrupts
> are emulated. So GVT-g doesn't need i915 host pass it these interrupts. For now, it seems GVT-g only
> needs i915 host forwards some GT interrupts used to emulate the status of guest workload execustion:
>  
> - PIPE_CONTROL_NOTIFY used by Windows guest
> - MI_USER_INTERRUPT used by Linux guest
>  
> Let these two interrupt always be enabled under i915 host mode.
>  
> - CONTEXT SWITCH interrupt
>  
> This interrupt has been always enablined under i915 host mode. So no extra modification is needed.
> 
> Then call vgt interrupt event forwarding handler at the end of gen8_irq_handler.
>  
> Maybe this should be simplest design.
>  
> Welcome to discuss!

Imo the simplest approach is to just refcount users for each interrupt.
And then GVT-g can simply grab a reference for each guest, like i915 host
code already does. An example would be the ring MI_USER_NOTIFY interrupt.
For that we have two main functions:

intel_ring_buffer->irq_get() and intel_ring_buffer->irq_put(). Currently
there's only one user for this in __i915_wait_request, but note that this
one is concurrent (you can call this function without holding
dev->struct_mutex). The ->irq_get/put() implementations have internally a
refcount to make sure the interrupt itself will get enabled for the first
user and only disabled once the last user stops.

For vblanks we have something similar in drm_irq.c with
drm_vblank_get/put(). That's imo how interrupt enabling should be
coordinated between gvt and i915 host code - gvt code should just use the
correct irq_get/put() functions for each client vm.

The other part is handling such interrupts, and for those I think the
simplest approach is to insert a callback into gvt code from the
i915_irq.c irq handler. For MI_NOTIFY_USER that would be notify_ring
(which is btw violating our naming convention that all interrupt handlers
should have an _irq_handler suffic). And for vblanks it would be
intel_pipe_vblank_handle. For other interrupts there's similar things and
for some we might need to create new infrastructure right away.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
ohttp://blog.ffwll.ch


More information about the Intel-gfx mailing list