[PATCH 1/3] drm/radeon: GPU virtual memory support v22

[Daniel Vetter]

On Mon, Jan 09, 2012 at 10:44:15AM -0500, Jerome Glisse wrote:
> On Mon, Jan 09, 2012 at 09:31:16AM +0100, Daniel Vetter wrote:
> > On Sun, Jan 08, 2012 at 05:56:31PM -0500, Jerome Glisse wrote:
> > > On Sun, Jan 8, 2012 at 9:05 AM, Daniel Vetter <daniel at ffwll.ch> wrote:
> > > > Hi all,
> > > >
> > > > Meh, I've wanted to port the small set of helpers nouveau already has to
> > > > handle per open fd gpu virtual address spaces to core drm, so that I could
> > > > reuse them for i915. Just to go one small step towards unifying drivers in
> > > > drm/* a bit ...
> > > >
> > > > Looks like I'll have another driver to wrestle or just forget about it and
> > > > reinvet that wheel for i915, too.
> > > >
> > > > </slight rant>
> > > >
> > > > Cheers, Daniel
> > > > --
> > > 
> > > I looked at nouveau before writing this code, thing is, in the end
> > > there is little common code especialy when you take different path on
> > > how you handle things (persistent or dynamic page table for instance).
> > > Thought couple things can still be share. Note that the whole radeon
> > > code is designed with the possibility of having several address space
> > > per process, thought there is no use for such things today we believe
> > > things like opencl+opengl can benefit of each having their own address
> > > space.
> > 
> > - I've realized when looking through nouveau that we likely can't share
> >   match more than a gem_bo->vma lookup plus a bunch of helper functions.
> > 
> > - Imo having more than one gpu virtual address space per fd doesn't make
> >   much sense. libdrm (at least for i915) is mostly just about issueing
> >   cmdbuffers. Hence it's probably easier to just open two fds and
> >   instantiate two libdrm buffer managers if you want two address spaces
> >   for otherwise you have to teach libdrm that the same buffer object still
> >   can have different addresses (which is pretty much against the point of
> >   gpu virtual address spaces).
> 
> Radeon barely use libdrm (only the ddx use part of it).

Ok, that explains things a bit.

> > I also realize that in the dri1 days there's been way too much common code
> > that only gets used by one or two drivers and hence isn't really commonly
> > useable at all (and also not really of decent quality). So I'm all in
> > favour for driver-specific stuff, especially for execution and memory
> > management. But:
> > 
> > - nouveau already has gpu virtual address spaces, radeon just grew them
> >   with this patch and i915 is on track to get them, too: Patches to enable
> >   the different hw addressing mode for Sandybridge and later are ready,
> >   and with Ivybridge hw engineers kinked out the remaining bugs so we can
> >   actually context-switch between different address spaces without hitting
> >   hw bugs.
> > 
> > - The more general picture is that with the advent of more general-purpose
> >   apis and usecases for gpus like opencl (or also background video
> >   encoding/decoding/transcoding with libva) users will want to control gpu
> >   resources. So I expect that we'll grow resource limits, schedulers with
> >   priorities and maybe also something like control groups in a few years.
> >   But if we don't put a bit of thought into the commonalities of things
> >   like gpu virtual address spaces, scheduling and similar things I fear we
> >   won't be able to create a sensible common interface to allocate and
> >   control resources in the feature. Which will result in a sub-par
> >   experience. 
> 
> I wish we could come up with a common API for different GPU but the fact
> is all kernel API we exposed so far is specific to each GPU beside
> modesetting. Anythings that use the processing power of GPU use
> dedicated API.
> 
> For instance radeon virtual address give control to userspace, ie userspace
> decide the virtual address space at which each bo will be. On the contrary
> nouveau do the allocation in the kernel. We did choose userspace to be in
> charge for few reasons (top of my head) :
> - allow to have 1:1 mapping btw cpu address space and gpu without having
>   playing with process vm in the kernel
> - allow easy command stream replay (no need to rewrite the command
>   stream because virtual address are different)

This stuff here is awesome and I've really missed it from the changelog. I
think the value in having a discussion across drivers isn't always in
coming up with a common interface, but sometimes just in learning each
anothers design tradeoffs.

> Scheduling is next big things coming up, with lastest gen of gpu growing
> the capacity to preempt gpu task and also offering a finer control over
> the GPU cores. Again here i fear that we will each grow our own API.
> 
> My believe is that as long as we expose a low level API to use the GPU
> there is no way we can provide a common API for things like scheduling
> or virtual address space. The way you submit command is tightly related
> to scheduling and virtual address stuff.

Well I think for the case of virtual memory the following imo makes sense
- Some core notion of an address space, I'd have done a fixed 1:1 mapping
  with open fds, but I see the reasons to allow more flexible.
- Some core to bind/unbind bos into a specific address space. The driver
  would be in full control of where an object is place because there's no
  way we can accomodate for all hw constrains and (like for readoen with
  userspace managed vms) sw design decisions. Core would essentially just
  track things.
- Some simple way to go from a (gem, vm) pair to the corresponding gpu vma
  that tells you the gpu virtual address (and perhaps other things).

Afaics all three drivers could use that and this simple thing is already
good enough to export an abstraction of gpu virtual address spaces good
enough for resource management. E.g. you could limit virtual address space
usage in a per-process fashion by allowing the drm core
bind_to_gpu_address_space function to disallow a binding a bo if it would
put total usage over the limit. Adding sligthly more information
(something like ttm placement should be good enough) and we have
full-blown rss tracking and resource limiting like the memory cgroup
controller allows for system memory.

So yes I beg to differ, I think we _can_ get pretty usefull stuff with a
really minimal set of common core code. Obviously all the low-level stuff
will still be driver-specific, and we'll always see driver-specific
extensions.

Cheers, Daniel
-- 
Daniel Vetter
Mail: daniel at ffwll.ch
Mobile: +41 (0)79 365 57 48