[Intel-gfx] About the iGVT-g's requirement to pin guest contexts in VM

[Daniel Vetter]

On Wed, Sep 02, 2015 at 05:20:34PM +0800, Zhiyuan Lv wrote:
> Hi Daniel,
> 
> Thanks for the comments! And my reply in line:
> 
> On Wed, Sep 02, 2015 at 10:19:03AM +0200, Daniel Vetter wrote:
> > > > 
> > > > Also you obviously have to complete the copying from shadow->guest ctx
> > > > before you send the irq to the guest to signal ctx completion. Which means
> > > > there's really no overall problem here from a design pov, the only thing
> > > 
> > > Right. We cannot control when guest driver sees seqno change, but we
> > > can control when guest sees context interrupts. The guest CSB update
> > > and interrupt injection will be after we finish writing guest
> > > contexts.
> > > 
> > > So right now we have two options of context shadowing: one is to track
> > > the whole life-cycle of guest context, and another is to do the shadow
> > > work in context schedule-in/schedule-out time. Zhi draws a nice
> > > picture of them.
> > > 
> > > Currently we do not have concrete performance comparison of the two
> > > approaches. We will have a try and see. And about this patchset, I
> > > will remove the "context notification" part and send out an updated
> > > version. Thanks!
> > > 
> > > > you have to do is fix up bugs in the host code (probably you should just
> > > > write through the ggtt).
> > > 
> > > Sorry could you elaborate a little more about this? Guest context may
> > > not always be in aperture right?
> > 
> > Yeah the high-level problem is that global gtt is contended (we already
> > have trouble with that on xengt and there's the ongoing but unfished
> > partial mmap support for that). And permanently pinning execlist contexts
> > will cause lots of troubles.
> > 
> > Windows can do this because it segments the global gtt into different
> > parts (at least last time I looked at their memory manager), which means
> > execlist will never sit in the middle of the range used for mmaps. But
> > linux has a unified memory manager, which means execlist can sit anywhere,
> > and therefore badly fragment the global gtt. If we pin them then that will
> > cause trouble after long system uptime. And afaiui xengt is mostly aimed
> > at servers, where the uptime assumption should be "runs forever".
> 
> In server side, we do not expect host to run much graphics workloads
> (all should be in virtual machines with shorter uptime). But yeah, gm
> fragmentation is still an issue. Thanks for the explanation!

Fragmentation is a lot worse on the guest side (due to the reduce global
gtt space due to ballooning). Host isn't really any different.

> > Compounding factor is that despite that I raised this in the original
> > review execlist is still not yet using the active list in upstream and
> > instead does short-time pinning. It's better than pinning forever but
> > still breaks the eviction logic.
> > 
> > What Chris Wilson and I talked about forever is adding an object-specific
> > global_gtt_unbind hook. The idea is that this would be called when
> > unbinding/evicting a special object (e.g. hw context), and you could use
> > that to do the host signalling. That would be the perfect combination of
> > both approaches:
> 
> Thanks for the suggestion! That sounds great! Right now in XenGT part
> we still plan to try the shadow context work in context
> schedule-in/out time. With this way, we do not need to pin context as
> well as the host notification. We will collect some performance data
> to see how it works.
> 
> I sent out v2 patch set which has removed the pin/unpin of execlist
> contexts. The patchset addressed review comments from you, Chris and
> Joonas(Sorry I forgot to add CC to related people). Is that patch set
> OK to be merged? Thanks!

I'll defer to detailed reviewers, but if the static pinning is gone then
I'm ok. We can add the optimization I described here later on.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch