[PATCH 1/3] drm/radeon: GPU virtual memory support v22

[Alex Deucher]

On Mon, Jan 9, 2012 at 10:44 AM, Jerome Glisse <j.glisse at gmail.com> wrote:
> On Mon, Jan 09, 2012 at 09:31:16AM +0100, Daniel Vetter wrote:
>> On Sun, Jan 08, 2012 at 05:56:31PM -0500, Jerome Glisse wrote:
>> > On Sun, Jan 8, 2012 at 9:05 AM, Daniel Vetter <daniel at ffwll.ch> wrote:
>> > > Hi all,
>> > >
>> > > Meh, I've wanted to port the small set of helpers nouveau already has to
>> > > handle per open fd gpu virtual address spaces to core drm, so that I could
>> > > reuse them for i915. Just to go one small step towards unifying drivers in
>> > > drm/* a bit ...
>> > >
>> > > Looks like I'll have another driver to wrestle or just forget about it and
>> > > reinvet that wheel for i915, too.
>> > >
>> > > </slight rant>
>> > >
>> > > Cheers, Daniel
>> > > --
>> >
>> > I looked at nouveau before writing this code, thing is, in the end
>> > there is little common code especialy when you take different path on
>> > how you handle things (persistent or dynamic page table for instance).
>> > Thought couple things can still be share. Note that the whole radeon
>> > code is designed with the possibility of having several address space
>> > per process, thought there is no use for such things today we believe
>> > things like opencl+opengl can benefit of each having their own address
>> > space.
>>
>> - I've realized when looking through nouveau that we likely can't share
>>   match more than a gem_bo->vma lookup plus a bunch of helper functions.
>>
>> - Imo having more than one gpu virtual address space per fd doesn't make
>>   much sense. libdrm (at least for i915) is mostly just about issueing
>>   cmdbuffers. Hence it's probably easier to just open two fds and
>>   instantiate two libdrm buffer managers if you want two address spaces
>>   for otherwise you have to teach libdrm that the same buffer object still
>>   can have different addresses (which is pretty much against the point of
>>   gpu virtual address spaces).
>
> Radeon barely use libdrm (only the ddx use part of it).
>
>>
>> I also realize that in the dri1 days there's been way too much common code
>> that only gets used by one or two drivers and hence isn't really commonly
>> useable at all (and also not really of decent quality). So I'm all in
>> favour for driver-specific stuff, especially for execution and memory
>> management. But:
>>
>> - nouveau already has gpu virtual address spaces, radeon just grew them
>>   with this patch and i915 is on track to get them, too: Patches to enable
>>   the different hw addressing mode for Sandybridge and later are ready,
>>   and with Ivybridge hw engineers kinked out the remaining bugs so we can
>>   actually context-switch between different address spaces without hitting
>>   hw bugs.
>>
>> - The more general picture is that with the advent of more general-purpose
>>   apis and usecases for gpus like opencl (or also background video
>>   encoding/decoding/transcoding with libva) users will want to control gpu
>>   resources. So I expect that we'll grow resource limits, schedulers with
>>   priorities and maybe also something like control groups in a few years.
>>   But if we don't put a bit of thought into the commonalities of things
>>   like gpu virtual address spaces, scheduling and similar things I fear we
>>   won't be able to create a sensible common interface to allocate and
>>   control resources in the feature. Which will result in a sub-par
>>   experience.
>
> I wish we could come up with a common API for different GPU but the fact
> is all kernel API we exposed so far is specific to each GPU beside
> modesetting. Anythings that use the processing power of GPU use
> dedicated API.
>
> For instance radeon virtual address give control to userspace, ie userspace
> decide the virtual address space at which each bo will be. On the contrary
> nouveau do the allocation in the kernel. We did choose userspace to be in
> charge for few reasons (top of my head) :
> - allow to have 1:1 mapping btw cpu address space and gpu without having
>  playing with process vm in the kernel
> - allow easy command stream replay (no need to rewrite the command
>  stream because virtual address are different)
>
> Scheduling is next big things coming up, with lastest gen of gpu growing
> the capacity to preempt gpu task and also offering a finer control over
> the GPU cores. Again here i fear that we will each grow our own API.
>
> My believe is that as long as we expose a low level API to use the GPU
> there is no way we can provide a common API for things like scheduling
> or virtual address space. The way you submit command is tightly related
> to scheduling and virtual address stuff.
>
>> But if my google-fu doesn't fail me gpu address spaces for radeon was
>> posted the first time as v22 ever on a public list and merged right away,
>> so there's been simply no time to discuss cross-driver issues.  Which is
>> part of why I'm slightly miffed ;-)
>>
>
> I too wish that we could have release this earlier. I guess it was included
> because we still managed to get enough eyes familiar with radeon to look and
> play with this code.

We would have liked to as well, but with the holidays and internal
review process and people being on vacation, and the kernel merge
window timing it didn't work out this time.  Still as Jerome said, our
VM setup is tightly coupled with the way our HW works.  I'm not sure
how much code could actually be shared.

Alex

>
> Cheers,
> Jerome