[RFC] Mechanism for high priority scheduling in amdgpu

Pierre-Loup Griffais pgriffais at valvesoftware.com
Mon Dec 19 15:49:53 UTC 2016 

On Dec 19, 2016 6:48 AM, Serguei Sagalovitch <serguei.sagalovitch at amd.com> wrote:
> If compute queue is occupied only by you, the efficiency
 > is equal with setting job queue to high priority I think.
The only risk is the situation when graphics will take all
needed CUs. But in any case it should be very good test.

Andres/Pierre-Loup,

Did you try to do it or it is a lot of work for you?

The system will be fully loaded by the VR client application when this feature will need to be used, with hopefully both a graphics and compute job in flight using 100% of the CU capacity.

Let me try to succintly sum up requirements since you asked in the other branch:

On a fully loaded system (optimal occupancy by VR client app), we would like the VR runtime to be able to submit a task (graphics or compute, but we realize only compute might be possible for best results) and get results in a consistent amount of time. Ideally that time would be close to the time it would take to complete the same task on an otherwise idle system, but it's assumed there would be a fixed cost added to it due to winding down in-flight CUs. The quality of service provided by the feature would depend on how predictably small such a cost would be. 11ms would be a current upper limit but not really a useful number for the purpose of discussion, as the feature would be beyond useless at that point. Being able to intervene 1ms before vblank of the HMD and consistently get our task complete in time would be good.


BTW: If there is  non-VR application which will use high-priority
h/w queue then VR application will suffer.  Any ideas how
to solve it?

The intent is that this interface will require some sort of privilege that only the VR compositor would have on a well-configured VR system, higher-than-average niceness would be one idea. If you have any suggestions there also, it would be good discussion.

Thanks,
 - Pierre-Loup


Sincerely yours,
Serguei Sagalovitch

On 2016-12-19 12:50 AM, zhoucm1 wrote:
> Do you encounter the priority issue for compute queue with current
> driver?
>
> If compute queue is occupied only by you, the efficiency is equal with
> setting job queue to high priority I think.
>
> Regards,
> David Zhou
>
> On 2016年12月19日 13:29, Andres Rodriguez wrote:
>> Yes, vulkan is available on all-open through the mesa radv UMD.
>>
>> I'm not sure if I'm asking for too much, but if we can coordinate a
>> similar interface in radv and amdgpu-pro at the vulkan level that
>> would be great.
>>
>> I'm not sure what that's going to be yet.
>>
>> - Andres
>>
>> On 12/19/2016 12:11 AM, zhoucm1 wrote:
>>>
>>>
>>> On 2016年12月19日 11:33, Pierre-Loup A. Griffais wrote:
>>>> We're currently working with the open stack; I assume that a
>>>> mechanism could be exposed by both open and Pro Vulkan userspace
>>>> drivers and that the amdgpu kernel interface improvements we would
>>>> pursue following this discussion would let both drivers take
>>>> advantage of the feature, correct?
>>> Of course.
>>> Does open stack have Vulkan support?
>>>
>>> Regards,
>>> David Zhou
>>>>
>>>> On 12/18/2016 07:26 PM, zhoucm1 wrote:
>>>>> By the way, are you using all-open driver or amdgpu-pro driver?
>>>>>
>>>>> +David Mao, who is working on our Vulkan driver.
>>>>>
>>>>> Regards,
>>>>> David Zhou
>>>>>
>>>>> On 2016年12月18日 06:05, Pierre-Loup A. Griffais wrote:
>>>>>> Hi Serguei,
>>>>>>
>>>>>> I'm also working on the bringing up our VR runtime on top of amgpu;
>>>>>> see replies inline.
>>>>>>
>>>>>> On 12/16/2016 09:05 PM, Sagalovitch, Serguei wrote:
>>>>>>> Andres,
>>>>>>>
>>>>>>>>  For current VR workloads we have 3 separate processes running
>>>>>>>> actually:
>>>>>>> So we could have potential memory overcommit case or do you do
>>>>>>> partitioning
>>>>>>> on your own?  I would think that there is need to avoid
>>>>>>> overcomit in
>>>>>>> VR case to
>>>>>>> prevent any BO migration.
>>>>>>
>>>>>> You're entirely correct; currently the VR runtime is setting up
>>>>>> prioritized CPU scheduling for its VR compositor, we're working on
>>>>>> prioritized GPU scheduling and pre-emption (eg. this thread), and in
>>>>>> the future it will make sense to do work in order to make sure that
>>>>>> its memory allocations do not get evicted, to prevent any unwelcome
>>>>>> additional latency in the event of needing to perform just-in-time
>>>>>> reprojection.
>>>>>>
>>>>>>> BTW: Do you mean __real__ processes or threads?
>>>>>>> Based on my understanding sharing BOs between different processes
>>>>>>> could introduce additional synchronization constrains. btw: I am
>>>>>>> not
>>>>>>> sure
>>>>>>> if we are able to share Vulkan sync. object cross-process boundary.
>>>>>>
>>>>>> They are different processes; it is important for the compositor
>>>>>> that
>>>>>> is responsible for quality-of-service features such as consistently
>>>>>> presenting distorted frames with the right latency, reprojection,
>>>>>> etc,
>>>>>> to be separate from the main application.
>>>>>>
>>>>>> Currently we are using unreleased cross-process memory and semaphore
>>>>>> extensions to fetch updated eye images from the client application,
>>>>>> but the just-in-time reprojection discussed here does not actually
>>>>>> have any direct interactions with cross-process resource sharing,
>>>>>> since it's achieved by using whatever is the latest, most up-to-date
>>>>>> eye images that have already been sent by the client application,
>>>>>> which are already available to use without additional
>>>>>> synchronization.
>>>>>>
>>>>>>>
>>>>>>>>    3) System compositor (we are looking at approaches to remove
>>>>>>>> this
>>>>>>>> overhead)
>>>>>>> Yes,  IMHO the best is to run in  "full screen mode".
>>>>>>
>>>>>> Yes, we are working on mechanisms to present directly to the headset
>>>>>> display without any intermediaries as a separate effort.
>>>>>>
>>>>>>>
>>>>>>>>  The latency is our main concern,
>>>>>>> I would assume that this is the known problem (at least for compute
>>>>>>> usage).
>>>>>>> It looks like that amdgpu / kernel submission is rather CPU
>>>>>>> intensive
>>>>>>> (at least
>>>>>>> in the default configuration).
>>>>>>
>>>>>> As long as it's a consistent cost, it shouldn't an issue.
>>>>>> However, if
>>>>>> there's high degrees of variance then that would be troublesome
>>>>>> and we
>>>>>> would need to account for the worst case.
>>>>>>
>>>>>> Hopefully the requirements and approach we described make sense,
>>>>>> we're
>>>>>> looking forward to your feedback and suggestions.
>>>>>>
>>>>>> Thanks!
>>>>>>  - Pierre-Loup
>>>>>>
>>>>>>>
>>>>>>> Sincerely yours,
>>>>>>> Serguei Sagalovitch
>>>>>>>
>>>>>>>
>>>>>>> From: Andres Rodriguez <andresr at valvesoftware.com>
>>>>>>> Sent: December 16, 2016 10:00 PM
>>>>>>> To: Sagalovitch, Serguei; amd-gfx at lists.freedesktop.org
>>>>>>> Subject: RE: [RFC] Mechanism for high priority scheduling in amdgpu
>>>>>>>
>>>>>>> Hey Serguei,
>>>>>>>
>>>>>>>> [Serguei] No. I mean pipe :-) as MEC define it.  As far as I
>>>>>>>> understand (by simplifying)
>>>>>>>> some scheduling is per pipe.  I know about the current allocation
>>>>>>>> scheme but I do not think
>>>>>>>> that it is  ideal.  I would assume that we need  to switch to
>>>>>>>> dynamical partition
>>>>>>>> of resources  based on the workload otherwise we will have
>>>>>>>> resource
>>>>>>>> conflict
>>>>>>>> between Vulkan compute and  OpenCL.
>>>>>>>
>>>>>>> I agree the partitioning isn't ideal. I'm hoping we can start
>>>>>>> with a
>>>>>>> solution that assumes that
>>>>>>> only pipe0 has any work and the other pipes are idle (no HSA/ROCm
>>>>>>> running on the system).
>>>>>>>
>>>>>>> This should be more or less the use case we expect from VR users.
>>>>>>>
>>>>>>> I agree the split is currently not ideal, but I'd like to consider
>>>>>>> that a separate task, because
>>>>>>> making it dynamic is not straight forward :P
>>>>>>>
>>>>>>>> [Serguei] Vulkan works via amdgpu (kernel submissions) so amdkfd
>>>>>>>> will be not
>>>>>>>> involved.  I would assume that in the case of VR we will have
>>>>>>>> one main
>>>>>>>> application ("console" mode(?)) so we could temporally "ignore"
>>>>>>>> OpenCL/ROCm needs when VR is running.
>>>>>>>
>>>>>>> Correct, this is why we want to enable the high priority compute
>>>>>>> queue through
>>>>>>> libdrm-amdgpu, so that we can expose it through Vulkan later.
>>>>>>>
>>>>>>> For current VR workloads we have 3 separate processes running
>>>>>>> actually:
>>>>>>>     1) Game process
>>>>>>>     2) VR Compositor (this is the process that will require high
>>>>>>> priority queue)
>>>>>>>     3) System compositor (we are looking at approaches to remove
>>>>>>> this
>>>>>>> overhead)
>>>>>>>
>>>>>>> For now I think it is okay to assume no OpenCL/ROCm running
>>>>>>> simultaneously, but
>>>>>>> I would also like to be able to address this case in the future
>>>>>>> (cross-pipe priorities).
>>>>>>>
>>>>>>>> [Serguei]  The problem with pre-emption of graphics task:  (a) it
>>>>>>>> may take time so
>>>>>>>> latency may suffer
>>>>>>>
>>>>>>> The latency is our main concern, we want something that is
>>>>>>> predictable. A good
>>>>>>> illustration of what the reprojection scheduling looks like can be
>>>>>>> found here:
>>>>>>> https://community.amd.com/servlet/JiveServlet/showImage/38-1310-104754/pastedImage_3.png
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> (b) to preempt we need to have different "context" - we want
>>>>>>>> to guarantee that submissions from the same context will be
>>>>>>>> executed
>>>>>>>> in order.
>>>>>>>
>>>>>>> This is okay, as the reprojection work doesn't have dependencies on
>>>>>>> the game context, and it
>>>>>>> even happens in a separate process.
>>>>>>>
>>>>>>>> BTW: (a) Do you want "preempt" and later resume or do you want
>>>>>>>> "preempt" and
>>>>>>>> "cancel/abort"
>>>>>>>
>>>>>>> Preempt the game with the compositor task and then resume it.
>>>>>>>
>>>>>>>> (b) Vulkan is generic API and could be used for graphics as
>>>>>>>> well as
>>>>>>>> for plain compute tasks (VK_QUEUE_COMPUTE_BIT).
>>>>>>>
>>>>>>> Yeah, the plan is to use vulkan compute. But if you figure out a
>>>>>>> way
>>>>>>> for us to get
>>>>>>> a guaranteed execution time using vulkan graphics, then I'll
>>>>>>> take you
>>>>>>> out for a beer :)
>>>>>>>
>>>>>>> Regards,
>>>>>>> Andres
>>>>>>> ________________________________________
>>>>>>> From: Sagalovitch, Serguei [Serguei.Sagalovitch at amd.com]
>>>>>>> Sent: Friday, December 16, 2016 9:13 PM
>>>>>>> To: Andres Rodriguez; amd-gfx at lists.freedesktop.org
>>>>>>> Subject: Re: [RFC] Mechanism for high priority scheduling in amdgpu
>>>>>>>
>>>>>>> Hi Andres,
>>>>>>>
>>>>>>> Please see inline (as [Serguei])
>>>>>>>
>>>>>>> Sincerely yours,
>>>>>>> Serguei Sagalovitch
>>>>>>>
>>>>>>>
>>>>>>> From: Andres Rodriguez <andresr at valvesoftware.com>
>>>>>>> Sent: December 16, 2016 8:29 PM
>>>>>>> To: Sagalovitch, Serguei; amd-gfx at lists.freedesktop.org
>>>>>>> Subject: RE: [RFC] Mechanism for high priority scheduling in amdgpu
>>>>>>>
>>>>>>> Hi Serguei,
>>>>>>>
>>>>>>> Thanks for the feedback. Answers inline as [AR].
>>>>>>>
>>>>>>> Regards,
>>>>>>> Andres
>>>>>>>
>>>>>>> ________________________________________
>>>>>>> From: Sagalovitch, Serguei [Serguei.Sagalovitch at amd.com]
>>>>>>> Sent: Friday, December 16, 2016 8:15 PM
>>>>>>> To: Andres Rodriguez; amd-gfx at lists.freedesktop.org
>>>>>>> Subject: Re: [RFC] Mechanism for high priority scheduling in amdgpu
>>>>>>>
>>>>>>> Andres,
>>>>>>>
>>>>>>>
>>>>>>> Quick comments:
>>>>>>>
>>>>>>> 1) To minimize "bubbles", etc. we need to "force" CU
>>>>>>> assignments/binding
>>>>>>> to high-priority queue  when it will be in use and "free" them
>>>>>>> later
>>>>>>> (we  do not want forever take CUs from e.g. graphic task to degrade
>>>>>>> graphics
>>>>>>> performance).
>>>>>>>
>>>>>>> Otherwise we could have scenario when long graphics task (or
>>>>>>> low-priority
>>>>>>> compute) will took all (extra) CUs and high--priority will wait for
>>>>>>> needed resources.
>>>>>>> It will not be visible on "NOP " but only when you submit "real"
>>>>>>> compute task
>>>>>>> so I would recommend  not to use "NOP" packets at all for testing.
>>>>>>>
>>>>>>> It (CU assignment) could be relatively easy done when everything is
>>>>>>> going via kernel
>>>>>>> (e.g. as part of frame submission) but I must admit that I am
>>>>>>> not sure
>>>>>>> about the best way for user level submissions (amdkfd).
>>>>>>>
>>>>>>> [AR] I wasn't aware of this part of the programming sequence.
>>>>>>> Thanks
>>>>>>> for the heads up!
>>>>>>> Is this similar to the CU masking programming?
>>>>>>> [Serguei] Yes. To simplify: the problem is that "scheduler" when
>>>>>>> deciding which
>>>>>>> queue to  run will check if there is enough resources and if not
>>>>>>> then
>>>>>>> it will begin
>>>>>>> to check other queues with lower priority.
>>>>>>>
>>>>>>> 2) I would recommend to dedicate the whole pipe to high-priority
>>>>>>> queue and have
>>>>>>> nothing their except it.
>>>>>>>
>>>>>>> [AR] I'm guessing in this context you mean pipe = queue? (as
>>>>>>> opposed
>>>>>>> to the MEC definition
>>>>>>> of pipe, which is a grouping of queues). I say this because amdgpu
>>>>>>> only has access to 1 pipe,
>>>>>>> and the rest are statically partitioned for amdkfd usage.
>>>>>>>
>>>>>>> [Serguei] No. I mean pipe :-)  as MEC define it.  As far as I
>>>>>>> understand (by simplifying)
>>>>>>> some scheduling is per pipe.  I know about the current allocation
>>>>>>> scheme but I do not think
>>>>>>> that it is  ideal.  I would assume that we need  to switch to
>>>>>>> dynamical partition
>>>>>>> of resources  based on the workload otherwise we will have resource
>>>>>>> conflict
>>>>>>> between Vulkan compute and  OpenCL.
>>>>>>>
>>>>>>>
>>>>>>> BTW: Which user level API do you want to use for compute: Vulkan or
>>>>>>> OpenCL?
>>>>>>>
>>>>>>> [AR] Vulkan
>>>>>>>
>>>>>>> [Serguei] Vulkan works via amdgpu (kernel submissions) so amdkfd
>>>>>>> will
>>>>>>> be not
>>>>>>> involved.  I would assume that in the case of VR we will have
>>>>>>> one main
>>>>>>> application ("console" mode(?)) so we could temporally "ignore"
>>>>>>> OpenCL/ROCm needs when VR is running.
>>>>>>>
>>>>>>>>  we will not be able to provide a solution compatible with GFX
>>>>>>>> worloads.
>>>>>>> I assume that you are talking about graphics? Am I right?
>>>>>>>
>>>>>>> [AR] Yeah, my understanding is that pre-empting the currently
>>>>>>> running
>>>>>>> graphics job and scheduling in
>>>>>>> something else using mid-buffer pre-emption has some cases where it
>>>>>>> doesn't work well. But if with
>>>>>>> polaris10 it starts working well, it might be a better solution for
>>>>>>> us (because the whole reprojection
>>>>>>> work uses the vulkan graphics stack at the moment, and porting
>>>>>>> it to
>>>>>>> compute is not trivial).
>>>>>>>
>>>>>>> [Serguei]  The problem with pre-emption of graphics task: (a) it
>>>>>>> may
>>>>>>> take time so
>>>>>>> latency may suffer (b) to preempt we need to have different
>>>>>>> "context"
>>>>>>> - we want
>>>>>>> to guarantee that submissions from the same context will be
>>>>>>> executed
>>>>>>> in order.
>>>>>>> BTW: (a) Do you want  "preempt" and later resume or do you want
>>>>>>> "preempt" and
>>>>>>> "cancel/abort"?  (b) Vulkan is generic API and could be used
>>>>>>> for graphics as well as for plain compute tasks
>>>>>>> (VK_QUEUE_COMPUTE_BIT).
>>>>>>>
>>>>>>>
>>>>>>> Sincerely yours,
>>>>>>> Serguei Sagalovitch
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From: amd-gfx <amd-gfx-bounces at lists.freedesktop.org> on behalf of
>>>>>>> Andres Rodriguez <andresr at valvesoftware.com>
>>>>>>> Sent: December 16, 2016 6:15 PM
>>>>>>> To: amd-gfx at lists.freedesktop.org
>>>>>>> Subject: [RFC] Mechanism for high priority scheduling in amdgpu
>>>>>>>
>>>>>>> Hi Everyone,
>>>>>>>
>>>>>>> This RFC is also available as a gist here:
>>>>>>> https://gist.github.com/lostgoat/7000432cd6864265dbc2c3ab93204249
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [RFC] Mechanism for high priority scheduling in amdgpu
>>>>>>> gist.github.com
>>>>>>> [RFC] Mechanism for high priority scheduling in amdgpu
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [RFC] Mechanism for high priority scheduling in amdgpu
>>>>>>> gist.github.com
>>>>>>> [RFC] Mechanism for high priority scheduling in amdgpu
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [RFC] Mechanism for high priority scheduling in amdgpu
>>>>>>> gist.github.com
>>>>>>> [RFC] Mechanism for high priority scheduling in amdgpu
>>>>>>>
>>>>>>>
>>>>>>> We are interested in feedback for a mechanism to effectively
>>>>>>> schedule
>>>>>>> high
>>>>>>> priority VR reprojection tasks (also referred to as
>>>>>>> time-warping) for
>>>>>>> Polaris10
>>>>>>> running on the amdgpu kernel driver.
>>>>>>>
>>>>>>> Brief context:
>>>>>>> --------------
>>>>>>>
>>>>>>> The main objective of reprojection is to avoid motion sickness
>>>>>>> for VR
>>>>>>> users in
>>>>>>> scenarios where the game or application would fail to finish
>>>>>>> rendering a new
>>>>>>> frame in time for the next VBLANK. When this happens, the user's
>>>>>>> head
>>>>>>> movements
>>>>>>> are not reflected on the Head Mounted Display (HMD) for the
>>>>>>> duration
>>>>>>> of an
>>>>>>> extra frame. This extended mismatch between the inner ear and the
>>>>>>> eyes may
>>>>>>> cause the user to experience motion sickness.
>>>>>>>
>>>>>>> The VR compositor deals with this problem by fabricating a new
>>>>>>> frame
>>>>>>> using the
>>>>>>> user's updated head position in combination with the previous
>>>>>>> frames.
>>>>>>> This
>>>>>>> avoids a prolonged mismatch between the HMD output and the inner
>>>>>>> ear.
>>>>>>>
>>>>>>> Because of the adverse effects on the user, we require high
>>>>>>> confidence that the
>>>>>>> reprojection task will complete before the VBLANK interval. Even if
>>>>>>> the GFX pipe
>>>>>>> is currently full of work from the game/application (which is most
>>>>>>> likely the case).
>>>>>>>
>>>>>>> For more details and illustrations, please refer to the following
>>>>>>> document:
>>>>>>> https://community.amd.com/community/gaming/blog/2016/03/28/asynchronous-shaders-evolved
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Gaming: Asynchronous Shaders Evolved | Community
>>>>>>> community.amd.com
>>>>>>> One of the most exciting new developments in GPU technology over
>>>>>>> the
>>>>>>> past year has been the adoption of asynchronous shaders, which can
>>>>>>> make more efficient use of ...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Gaming: Asynchronous Shaders Evolved | Community
>>>>>>> community.amd.com
>>>>>>> One of the most exciting new developments in GPU technology over
>>>>>>> the
>>>>>>> past year has been the adoption of asynchronous shaders, which can
>>>>>>> make more efficient use of ...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Gaming: Asynchronous Shaders Evolved | Community
>>>>>>> community.amd.com
>>>>>>> One of the most exciting new developments in GPU technology over
>>>>>>> the
>>>>>>> past year has been the adoption of asynchronous shaders, which can
>>>>>>> make more efficient use of ...
>>>>>>>
>>>>>>>
>>>>>>> Requirements:
>>>>>>> -------------
>>>>>>>
>>>>>>> The mechanism must expose the following functionaility:
>>>>>>>
>>>>>>>     * Job round trip time must be predictable, from submission to
>>>>>>> fence signal
>>>>>>>
>>>>>>>     * The mechanism must support compute workloads.
>>>>>>>
>>>>>>> Goals:
>>>>>>> ------
>>>>>>>
>>>>>>>     * The mechanism should provide low submission latencies
>>>>>>>
>>>>>>> Test: submitting a NOP packet through the mechanism on busy
>>>>>>> hardware
>>>>>>> should
>>>>>>> be equivalent to submitting a NOP on idle hardware.
>>>>>>>
>>>>>>> Nice to have:
>>>>>>> -------------
>>>>>>>
>>>>>>>     * The mechanism should also support GFX workloads.
>>>>>>>
>>>>>>> My understanding is that with the current hardware capabilities in
>>>>>>> Polaris10 we
>>>>>>> will not be able to provide a solution compatible with GFX
>>>>>>> worloads.
>>>>>>>
>>>>>>> But I would love to hear otherwise. So if anyone has an idea,
>>>>>>> approach or
>>>>>>> suggestion that will also be compatible with the GFX ring,
>>>>>>> please let
>>>>>>> us know
>>>>>>> about it.
>>>>>>>
>>>>>>>     * The above guarantees should also be respected by amdkfd
>>>>>>> workloads
>>>>>>>
>>>>>>> Would be good to have for consistency, but not strictly
>>>>>>> necessary as
>>>>>>> users running
>>>>>>> games are not traditionally running HPC workloads in the
>>>>>>> background.
>>>>>>>
>>>>>>> Proposed approach:
>>>>>>> ------------------
>>>>>>>
>>>>>>> Similar to the windows driver, we could expose a high priority
>>>>>>> compute queue to
>>>>>>> userspace.
>>>>>>>
>>>>>>> Submissions to this compute queue will be scheduled with high
>>>>>>> priority, and may
>>>>>>> acquire hardware resources previously in use by other queues.
>>>>>>>
>>>>>>> This can be achieved by taking advantage of the 'priority' field in
>>>>>>> the HQDs
>>>>>>> and could be programmed by amdgpu or the amdgpu scheduler. The
>>>>>>> relevant
>>>>>>> register fields are:
>>>>>>>         * mmCP_HQD_PIPE_PRIORITY
>>>>>>>         * mmCP_HQD_QUEUE_PRIORITY
>>>>>>>
>>>>>>> Implementation approach 1 - static partitioning:
>>>>>>> ------------------------------------------------
>>>>>>>
>>>>>>> The amdgpu driver currently controls 8 compute queues from
>>>>>>> pipe0. We can
>>>>>>> statically partition these as follows:
>>>>>>>         * 7x regular
>>>>>>>         * 1x high priority
>>>>>>>
>>>>>>> The relevant priorities can be set so that submissions to the high
>>>>>>> priority
>>>>>>> ring will starve the other compute rings and the GFX ring.
>>>>>>>
>>>>>>> The amdgpu scheduler will only place jobs into the high priority
>>>>>>> rings if the
>>>>>>> context is marked as high priority. And a corresponding priority
>>>>>>> should be
>>>>>>> added to keep track of this information:
>>>>>>>      * AMD_SCHED_PRIORITY_KERNEL
>>>>>>>      * -> AMD_SCHED_PRIORITY_HIGH
>>>>>>>      * AMD_SCHED_PRIORITY_NORMAL
>>>>>>>
>>>>>>> The user will request a high priority context by setting an
>>>>>>> appropriate flag
>>>>>>> in drm_amdgpu_ctx_in (AMDGPU_CTX_HIGH_PRIORITY or similar):
>>>>>>> https://github.com/torvalds/linux/blob/master/include/uapi/drm/amdgpu_drm.h#L163
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The setting is in a per context level so that we can:
>>>>>>>     * Maintain a consistent FIFO ordering of all submissions to a
>>>>>>> context
>>>>>>>     * Create high priority and non-high priority contexts in the
>>>>>>> same
>>>>>>> process
>>>>>>>
>>>>>>> Implementation approach 2 - dynamic priority programming:
>>>>>>> ---------------------------------------------------------
>>>>>>>
>>>>>>> Similar to the above, but instead of programming the priorities and
>>>>>>> amdgpu_init() time, the SW scheduler will reprogram the queue
>>>>>>> priorities
>>>>>>> dynamically when scheduling a task.
>>>>>>>
>>>>>>> This would involve having a hardware specific callback from the
>>>>>>> scheduler to
>>>>>>> set the appropriate queue priority: set_priority(int ring, int
>>>>>>> index,
>>>>>>> int priority)
>>>>>>>
>>>>>>> During this callback we would have to grab the SRBM mutex to
>>>>>>> perform
>>>>>>> the appropriate
>>>>>>> HW programming, and I'm not really sure if that is something we
>>>>>>> should be doing from
>>>>>>> the scheduler.
>>>>>>>
>>>>>>> On the positive side, this approach would allow us to program a
>>>>>>> range of
>>>>>>> priorities for jobs instead of a single "high priority" value",
>>>>>>> achieving
>>>>>>> something similar to the niceness API available for CPU scheduling.
>>>>>>>
>>>>>>> I'm not sure if this flexibility is something that we would need
>>>>>>> for
>>>>>>> our use
>>>>>>> case, but it might be useful in other scenarios (multiple users
>>>>>>> sharing compute
>>>>>>> time on a server).
>>>>>>>
>>>>>>> This approach would require a new int field in
>>>>>>> drm_amdgpu_ctx_in, or
>>>>>>> repurposing
>>>>>>> of the flags field.
>>>>>>>
>>>>>>> Known current obstacles:
>>>>>>> ------------------------
>>>>>>>
>>>>>>> The SQ is currently programmed to disregard the HQD priorities, and
>>>>>>> instead it picks
>>>>>>> jobs at random. Settings from the shader itself are also
>>>>>>> disregarded
>>>>>>> as this is
>>>>>>> considered a privileged field.
>>>>>>>
>>>>>>> Effectively we can get our compute wavefront launched ASAP, but we
>>>>>>> might not get the
>>>>>>> time we need on the SQ.
>>>>>>>
>>>>>>> The current programming would have to be changed to allow priority
>>>>>>> propagation
>>>>>>> from the HQD into the SQ.
>>>>>>>
>>>>>>> Generic approach for all HW IPs:
>>>>>>> --------------------------------
>>>>>>>
>>>>>>> For consistency purposes, the high priority context can be enabled
>>>>>>> for all HW IPs
>>>>>>> with support of the SW scheduler. This will function similarly
>>>>>>> to the
>>>>>>> current
>>>>>>> AMD_SCHED_PRIORITY_KERNEL priority, where the job can jump ahead of
>>>>>>> anything not
>>>>>>> commited to the HW queue.
>>>>>>>
>>>>>>> The benefits of requesting a high priority context for a
>>>>>>> non-compute
>>>>>>> queue will
>>>>>>> be lesser (e.g. up to 10s of wait time if a GFX command is stuck in
>>>>>>> front of
>>>>>>> you), but having the API in place will allow us to easily
>>>>>>> improve the
>>>>>>> implementation
>>>>>>> in the future as new features become available in new hardware.
>>>>>>>
>>>>>>> Future steps:
>>>>>>> -------------
>>>>>>>
>>>>>>> Once we have an approach settled, I can take care of the
>>>>>>> implementation.
>>>>>>>
>>>>>>> Also, once the interface is mostly decided, we can start
>>>>>>> thinking about
>>>>>>> exposing the high priority queue through radv.
>>>>>>>
>>>>>>> Request for feedback:
>>>>>>> ---------------------
>>>>>>>
>>>>>>> We aren't married to any of the approaches outlined above. Our goal
>>>>>>> is to
>>>>>>> obtain a mechanism that will allow us to complete the reprojection
>>>>>>> job within a
>>>>>>> predictable amount of time. So if anyone anyone has any
>>>>>>> suggestions for
>>>>>>> improvements or alternative strategies we are more than happy to
>>>>>>> hear
>>>>>>> them.
>>>>>>>
>>>>>>> If any of the technical information above is also incorrect, feel
>>>>>>> free to point
>>>>>>> out my misunderstandings.
>>>>>>>
>>>>>>> Looking forward to hearing from you.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Andres
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> amd-gfx mailing list
>>>>>>> amd-gfx at lists.freedesktop.org
>>>>>>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>>>>>>>
>>>>>>>
>>>>>>> amd-gfx Info Page - lists.freedesktop.org
>>>>>>> lists.freedesktop.org
>>>>>>> To see the collection of prior postings to the list, visit the
>>>>>>> amd-gfx Archives. Using amd-gfx: To post a message to all the list
>>>>>>> members, send email ...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> amd-gfx Info Page - lists.freedesktop.org
>>>>>>> lists.freedesktop.org
>>>>>>> To see the collection of prior postings to the list, visit the
>>>>>>> amd-gfx Archives. Using amd-gfx: To post a message to all the list
>>>>>>> members, send email ...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> amd-gfx mailing list
>>>>>>> amd-gfx at lists.freedesktop.org
>>>>>>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> amd-gfx mailing list
>>>>>> amd-gfx at lists.freedesktop.org
>>>>>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> amd-gfx mailing list
>>> amd-gfx at lists.freedesktop.org
>>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>>
>

Sincerely yours,
Serguei Sagalovitch


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/amd-gfx/attachments/20161219/29147a80/attachment-0001.html>

More information about the amd-gfx mailing list