Proposal for a DRM-compliant GPU command scheduler

[Daniel Vetter]

Hi,

I haven't attended xdc in chicago, but I've read through your slides and
looked a bit at the code. We're planing to implement gpu scheduling for
intel gpus, too, so I'm pretty interested in this area. Comments:

- your code seems to rather thightly integrated with how nvidia hw works.
  I'm not sure whether it's a good fit for other hw, especially if/when
  there's better support from the hw for context switching.

- by the looks of it, scheduling happens via: gpu completion irq handler
  -> realtime thread -> waking up of the blocked process that got put to
  sleep before command submission. There's also a fastpath that does not
  block the command submission if the scheduler allows the process to run
  immediately. I fear this has quite high overhead and the scheduler
  design doesn't seem to allow clever tricks like issuing gpu batchbuffers
  eagerly and patching up execution after the fact (if e.g. a previous
  batchbuffer used up too much time). Your argument that the scheduler
  completely disables itself is also a bit void - contemporary desktop and
  mobile systems always have mutliple clients: A compositor and the
  clients, sometimes there's even an X process rearing its ugly head ;-)

- Imo your code needs quite some clean-up. I've noticed e.g. that it
  doesn't use the linux struct list_head functions. It also seems to
  re-implement a waitqueue/completions in a (racy) way.

- Imo the area that would most benefit from a shared gpu scheduler
  infrastructure is the userspace interface, so that a common set of tools
  can be used accross different drivers. Your solutions to configure the
  scheduler seems to be to read a file in /etc from the kernel module
  which is ... a bit ugly, to say the least.

In short it'd be awesome if you can help in creating a gpu scheduler
infrastructure for linux. But unfortunately your current code is imo
pretty far away from something that could be merged.

Yours, Daniel

On Thu, Sep 15, 2011 at 04:20:13PM -0700, Shinpei KATO wrote:
> Hi,
> 
> I am the main developer of the TimeGraph GPU command scheduler, which was
> presented at XDC 2011 in Chicago a few days ago.
> Please let me propose this approach to scheduling GPU-accelerated processes
> with DRM.
> 
> This GPU scheduler will help to prioritize and isolate multiple
> GPU-accelerated processes executing concurrently for protecting important
> GPU workload in multi-tasking environments. It is designed and implemented
> at the DRM level as a "drm_sched" component. Each architecture-dependent
> driver (nouveau, radeon, i915, etc.) is also required to call the scheduling
> functions provided by this drm_sched component accordingly. Nothing needs to
> be changed in user-space runtimes.
> The priorities and resource limits can be specified through
> "/proc/driver/drm_sched/#PID/{sched_policy,resv_policy,priority,runtime,peri
> od}" and/or "/etc/drm_sched.spec". Other user interfaces could also be
> possible.
> The impact of prioritization and isolation on protecting important
> GPU-accelerated processes from competing GPU workload is quite significant
> (e.g., 3-D games can run at a 10x~ faster rate, using our GPU scheduler,
> when heavy workload is competing the GPU). Performance interference among
> GPU-accelerated processes can also be well-controlled.
> We can activate this GPU scheduler only when multiple processes use the GPU.
> Hence *nothing* would be harmful for a standalone execution.
> I felt that the audience at XDC 2011 was pretty supportive for this idea.
> 
> TimeGraph is developed in a collaborative project with Carnegie Mellon
> University, University of California Santa Cruz, and University of Tokyo.
> The project website is: http://rtml.ece.cmu.edu/projects/timegraph/
> 
> The documentation about how it works is:
> http://rtml.ece.cmu.edu/projects/timegraph/raw-attachment/wiki/documentation
> /drm_sched_rtgpu.pdf
> More information is available at:
> http://rtml.ece.cmu.edu/projects/timegraph/wiki/documentation
> 
> The instruction to install our Nouveau-based prototype driver is:
> http://rtml.ece.cmu.edu/projects/timegraph/wiki/install
> For convenience of development, the GPU scheduler is provided as an
> independent kernel module (https://gitorious.org/rtgpu/timegraph), but it
> can also be part of DRM.
> You will also need the Nouveau-tree Linux kernel patched for drm_sched
> (https://gitorious.org/rtgpu/linux-rtgpu). Please see the instruction above.
> There are not so many changes applied to the current kernel code, as you can
> quickly reference at:
> http://rtml.ece.cmu.edu/projects/timegraph/attachment/wiki/install/linux-rtg
> pu.patch
> 
> I would appreciate any comments and feedback from you.
> 
> Best Regards,
> - Shinpei Kato
> 
> _______________________________________________
> dri-devel mailing list
> dri-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/dri-devel

-- 
Daniel Vetter
Mail: daniel at ffwll.ch
Mobile: +41 (0)79 365 57 48