[PATCH 4/4] drm/amdgpu: MCBP based on DRM scheduler (v6)

Christian König christian.koenig at amd.com
Tue Sep 27 18:20:23 UTC 2022


This work is solely for gfx9 (e.g. Vega) and older.

Navi has a completely separate high priority gfx queue we can use for this.

Thanks,
Christian.

Am 27.09.22 um 19:49 schrieb Michel Dänzer:
> On 2022-09-27 08:06, Christian König wrote:
>> Hey Michel,
>>
>> JIadong is working on exposing high/low priority gfx queues for gfx9 and older hw generations by using mid command buffer preemption.
> Yeah, I've been keeping an eye on these patches. I'm looking forward to this working.
>
>
>> I know that you have been working on Gnome Mutter to make use from userspace for this. Do you have time to run some tests with that?
> I just tested the v8 series (first without amdgpu.mcbp=1 on the kernel command line, then with it, since I wasn't sure if it's needed) with https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.gnome.org%2FGNOME%2Fmutter%2F-%2Fmerge_requests%2F1880&data=05%7C01%7Cchristian.koenig%40amd.com%7Cc6345d9230004549ba4d08daa0b0abcd%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637998977913548768%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=P1Qo2AwDmfmPrxJe2SxTFsVjdJ9vjabK8s84ZVz%2Beh8%3D&reserved=0 on Navi 14.
>
> Unfortunately, I'm not seeing any change in behaviour. Even though mutter uses a high priority context via the EGL_IMG_context_priority extension, it's unable to reach a higher frame rate than a GPU-limited client[0]. The "Last preempted" line of /sys/kernel/debug/dri/0/amdgpu_fence_info remains at 0x00000000.
>
> Did I miss a step?
>
>
> [0] I used the GpuTest pixmark piano & plot3d benchmarks. With an Intel iGPU, mutter can achieve a higher frame rate than plot3d, though not than pixmark piano (presumably due to limited GPU preemption granularity).
>
>> Am 27.09.22 um 05:18 schrieb Zhu, Jiadong:
>>> [AMD Official Use Only - General]
>>>
>>>> I need more time for an in deep review of this, but form the one mile high view it looks correct to me now.
>>>> Can we do some pre-commit qa testing with this?
>>> I changed drm test "Command submission Test (GFX)" to send high priority ibs meanwhile running Manhattan on Screen/Unigine heaven foreground, checking mcbp/resubmit triggered by cat /sys/kernel/debug/dri/0/amdgpu_fence_info
>>>
>>> I have continued running this scenario for 2 daytime and 1 night, no hangs happen yet(lots of hangs has been fixed in the previous patches).
>>>
>>> I will ask QA team to do more test.
>>>
>>> Thanks,
>>> JIadong
>>>
>>> -----Original Message-----
>>> From: Christian König <ckoenig.leichtzumerken at gmail.com>
>>> Sent: Monday, September 26, 2022 2:49 PM
>>> To: Zhu, Jiadong <Jiadong.Zhu at amd.com>; amd-gfx at lists.freedesktop.org
>>> Cc: Tuikov, Luben <Luben.Tuikov at amd.com>; Koenig, Christian <Christian.Koenig at amd.com>; Grodzovsky, Andrey <Andrey.Grodzovsky at amd.com>
>>> Subject: Re: [PATCH 4/4] drm/amdgpu: MCBP based on DRM scheduler (v6)
>>>
>>> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
>>>
>>>
>>> Am 23.09.22 um 15:16 schrieb jiadong.zhu at amd.com:
>>>> From: "Jiadong.Zhu" <Jiadong.Zhu at amd.com>
>>>>
>>>> Trigger Mid-Command Buffer Preemption according to the priority of the
>>>> software rings and the hw fence signalling condition.
>>>>
>>>> The muxer saves the locations of the indirect buffer frames from the
>>>> software ring together with the fence sequence number in its fifo
>>>> queue, and pops out those records when the fences are signalled. The
>>>> locations are used to resubmit packages in preemption scenarios by coping the chunks from the software ring.
>>>>
>>>> v2: Update comment style.
>>>> v3: Fix conflict caused by previous modifications.
>>>> v4: Remove unnecessary prints.
>>>> v5: Fix corner cases for resubmission cases.
>>>> v6: Refactor functions for resubmission, calling fence_process in irq handler.
>>>>
>>>> Cc: Christian Koenig <Christian.Koenig at amd.com>
>>>> Cc: Luben Tuikov <Luben.Tuikov at amd.com>
>>>> Cc: Andrey Grodzovsky <Andrey.Grodzovsky at amd.com>
>>>> Acked-by: Luben Tuikov <luben.tuikov at amd.com>
>>>> Signed-off-by: Jiadong.Zhu <Jiadong.Zhu at amd.com>
>>> I need more time for an in deep review of this, but form the one mile high view it looks correct to me now.
>>>
>>> Can we do some pre-commit qa testing with this?
>>>
>>> Thanks,
>>> Christian.
>>>
>>>> ---
>>>>     drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c       |   2 +
>>>>     drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c     |  13 +
>>>>     drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h     |   3 +
>>>>     drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.c | 323 ++++++++++++++++---
>>>>     drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.h |  30 ++
>>>>     drivers/gpu/drm/amd/amdgpu/amdgpu_sw_ring.c  |  26 ++
>>>>     drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c       |   2 +
>>>>     drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c        |  10 +-
>>>>     8 files changed, 368 insertions(+), 41 deletions(-)
>>>>
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
>>>> index 258cffe3c06a..af86d87e2f3b 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
>>>> @@ -211,6 +211,7 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned num_ibs,
>>>>                 }
>>>>         }
>>>>
>>>> +     amdgpu_ring_ib_begin(ring);
>>>>         if (job && ring->funcs->init_cond_exec)
>>>>                 patch_offset = amdgpu_ring_init_cond_exec(ring);
>>>>
>>>> @@ -285,6 +286,7 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned num_ibs,
>>>>             ring->hw_prio == AMDGPU_GFX_PIPE_PRIO_HIGH)
>>>>                 ring->funcs->emit_wave_limit(ring, false);
>>>>
>>>> +     amdgpu_ring_ib_end(ring);
>>>>         amdgpu_ring_commit(ring);
>>>>         return 0;
>>>>     }
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
>>>> index 13db99d653bd..84b0b3c7d40f 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
>>>> @@ -33,6 +33,7 @@
>>>>
>>>>     #include <drm/amdgpu_drm.h>
>>>>     #include "amdgpu.h"
>>>> +#include "amdgpu_sw_ring.h"
>>>>     #include "atom.h"
>>>>
>>>>     /*
>>>> @@ -569,3 +570,15 @@ int amdgpu_ring_init_mqd(struct amdgpu_ring
>>>> *ring)
>>>>
>>>>         return mqd_mgr->init_mqd(adev, ring->mqd_ptr, &prop);
>>>>     }
>>>> +
>>>> +void amdgpu_ring_ib_begin(struct amdgpu_ring *ring) {
>>>> +     if (ring->is_sw_ring)
>>>> +             amdgpu_sw_ring_ib_begin(ring); }
>>>> +
>>>> +void amdgpu_ring_ib_end(struct amdgpu_ring *ring) {
>>>> +     if (ring->is_sw_ring)
>>>> +             amdgpu_sw_ring_ib_end(ring); }
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
>>>> index e90d327a589e..6fbc1627dab7 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
>>>> @@ -312,6 +312,9 @@ struct amdgpu_ring {
>>>>     #define amdgpu_ring_preempt_ib(r) (r)->funcs->preempt_ib(r)
>>>>
>>>>     int amdgpu_ring_alloc(struct amdgpu_ring *ring, unsigned ndw);
>>>> +void amdgpu_ring_ib_begin(struct amdgpu_ring *ring); void
>>>> +amdgpu_ring_ib_end(struct amdgpu_ring *ring);
>>>> +
>>>>     void amdgpu_ring_insert_nop(struct amdgpu_ring *ring, uint32_t count);
>>>>     void amdgpu_ring_generic_pad_ib(struct amdgpu_ring *ring, struct amdgpu_ib *ib);
>>>>     void amdgpu_ring_commit(struct amdgpu_ring *ring); diff --git
>>>> a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.c
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.c
>>>> index 662aadebf111..788567e3b743 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.c
>>>> @@ -28,23 +28,146 @@
>>>>
>>>>     #define AMDGPU_MUX_RESUBMIT_JIFFIES_TIMEOUT (HZ / 2)
>>>>
>>>> +static struct kmem_cache *amdgpu_mux_chunk_slab;
>>>> +
>>>> +static inline struct amdgpu_mux_entry *amdgpu_ring_mux_sw_entry(struct amdgpu_ring_mux *mux,
>>>> +                                                             struct
>>>> +amdgpu_ring *ring) {
>>>> +     return ring->entry_index < mux->ring_entry_size ?
>>>> +                     &mux->ring_entry[ring->entry_index] : NULL; }
>>>> +
>>>> +/* copy packages on sw ring range[begin, end) */ static void
>>>> +amdgpu_ring_mux_copy_pkt_from_sw_ring(struct amdgpu_ring_mux *mux,
>>>> +                                               struct amdgpu_ring *ring,
>>>> +                                               u64 s_start, u64
>>>> +s_end) {
>>>> +     u64 start, end;
>>>> +     struct amdgpu_ring *real_ring = mux->real_ring;
>>>> +
>>>> +     start = s_start & ring->buf_mask;
>>>> +     end = s_end & ring->buf_mask;
>>>> +
>>>> +     if (start == end) {
>>>> +             DRM_ERROR("no more data copied from sw ring\n");
>>>> +             return;
>>>> +     }
>>>> +     if (start > end) {
>>>> +             amdgpu_ring_alloc(real_ring, (ring->ring_size >> 2) + end - start);
>>>> +             amdgpu_ring_write_multiple(real_ring, (void *)&ring->ring[start],
>>>> +                                        (ring->ring_size >> 2) - start);
>>>> +             amdgpu_ring_write_multiple(real_ring, (void *)&ring->ring[0], end);
>>>> +     } else {
>>>> +             amdgpu_ring_alloc(real_ring, end - start);
>>>> +             amdgpu_ring_write_multiple(real_ring, (void *)&ring->ring[start], end - start);
>>>> +     }
>>>> +}
>>>> +
>>>> +static void amdgpu_mux_resubmit_chunks(struct amdgpu_ring_mux *mux,
>>>> +bool is_fallback) {
>>>> +     struct amdgpu_mux_entry *e = NULL;
>>>> +     struct amdgpu_mux_chunk *chunk;
>>>> +     uint32_t seq, last_seq;
>>>> +     int i;
>>>> +
>>>> +     if (is_fallback) {
>>>> +             if (!spin_trylock(&mux->lock)) {
>>>> +                     amdgpu_ring_mux_schedule_resubmit(mux);
>>>> +                     DRM_ERROR("reschedule resubmit\n");
>>>> +                     return;
>>>> +             }
>>>> +     } else {
>>>> +             spin_lock(&mux->lock);
>>>> +     }
>>>> +
>>>> +     /*find low priority entries:*/
>>>> +     if (!mux->s_resubmit) {
>>>> +             spin_unlock(&mux->lock);
>>>> +             return;
>>>> +     }
>>>> +
>>>> +     for (i = 0; i < mux->num_ring_entries; i++) {
>>>> +             if (mux->ring_entry[i].ring->hw_prio <= AMDGPU_RING_PRIO_DEFAULT) {
>>>> +                     e = &mux->ring_entry[i];
>>>> +                     break;
>>>> +             }
>>>> +     }
>>>> +
>>>> +     if (!e) {
>>>> +             DRM_ERROR("%s no low priority ring found\n", __func__);
>>>> +             spin_unlock(&mux->lock);
>>>> +             return;
>>>> +     }
>>>> +
>>>> +     last_seq = atomic_read(&e->ring->fence_drv.last_seq);
>>>> +     seq = mux->seqno_to_resubmit;
>>>> +     if (last_seq < seq) {
>>>> +             /*resubmit all the fences between (last_seq, seq]*/
>>>> +             list_for_each_entry(chunk, &e->list, entry) {
>>>> +                     if (chunk->sync_seq > last_seq && chunk->sync_seq <= seq) {
>>>> +                             amdgpu_ring_mux_copy_pkt_from_sw_ring(mux, e->ring,
>>>> +                                                                   chunk->start,
>>>> +                                                                   chunk->end);
>>>> +                             mux->wptr_resubmit = chunk->end;
>>>> +                             amdgpu_ring_commit(mux->real_ring);
>>>> +                     }
>>>> +             }
>>>> +     }
>>>> +
>>>> +     del_timer(&mux->resubmit_timer);
>>>> +     mux->s_resubmit = false;
>>>> +     spin_unlock(&mux->lock);
>>>> +}
>>>> +
>>>> +static void amdgpu_mux_resubmit_fallback(struct timer_list *t) {
>>>> +     struct amdgpu_ring_mux *mux = from_timer(mux, t,
>>>> +resubmit_timer);
>>>> +
>>>> +     DRM_INFO("calling %s\n", __func__);
>>>> +     amdgpu_mux_resubmit_chunks(mux, true); }
>>>> +
>>>>     int amdgpu_ring_mux_init(struct amdgpu_ring_mux *mux, struct amdgpu_ring *ring,
>>>>                          unsigned int entry_size)
>>>>     {
>>>>         mux->real_ring = ring;
>>>>         mux->num_ring_entries = 0;
>>>> +
>>>>         mux->ring_entry = kcalloc(entry_size, sizeof(struct amdgpu_mux_entry), GFP_KERNEL);
>>>>         if (!mux->ring_entry)
>>>>                 return -ENOMEM;
>>>>
>>>>         mux->ring_entry_size = entry_size;
>>>> +     mux->s_resubmit = false;
>>>> +
>>>> +     amdgpu_mux_chunk_slab = kmem_cache_create("amdgpu_mux_chunk",
>>>> +                                               sizeof(struct amdgpu_mux_chunk), 0,
>>>> +                                               SLAB_HWCACHE_ALIGN, NULL);
>>>> +     if (!amdgpu_mux_chunk_slab) {
>>>> +             DRM_ERROR("create amdgpu_mux_chunk cache failed\n");
>>>> +             return -ENOMEM;
>>>> +     }
>>>> +
>>>>         spin_lock_init(&mux->lock);
>>>> +     timer_setup(&mux->resubmit_timer, amdgpu_mux_resubmit_fallback,
>>>> + 0);
>>>>
>>>>         return 0;
>>>>     }
>>>>
>>>>     void amdgpu_ring_mux_fini(struct amdgpu_ring_mux *mux)
>>>>     {
>>>> +     struct amdgpu_mux_entry *e;
>>>> +     struct amdgpu_mux_chunk *chunk, *chunk2;
>>>> +     int i;
>>>> +
>>>> +     for (i = 0; i < mux->num_ring_entries; i++) {
>>>> +             e = &mux->ring_entry[i];
>>>> +             list_for_each_entry_safe(chunk, chunk2, &e->list, entry) {
>>>> +                     list_del(&chunk->entry);
>>>> +                     kmem_cache_free(amdgpu_mux_chunk_slab, chunk);
>>>> +             }
>>>> +     }
>>>> +     kmem_cache_destroy(amdgpu_mux_chunk_slab);
>>>>         kfree(mux->ring_entry);
>>>>         mux->ring_entry = NULL;
>>>>         mux->num_ring_entries = 0;
>>>> @@ -64,62 +187,46 @@ int amdgpu_ring_mux_add_sw_ring(struct amdgpu_ring_mux *mux, struct amdgpu_ring
>>>>         ring->entry_index = mux->num_ring_entries;
>>>>         e->ring = ring;
>>>>
>>>> +     INIT_LIST_HEAD(&e->list);
>>>>         mux->num_ring_entries += 1;
>>>>         return 0;
>>>>     }
>>>>
>>>> -static inline struct amdgpu_mux_entry *amdgpu_ring_mux_sw_entry(struct amdgpu_ring_mux *mux,
>>>> -                                                             struct amdgpu_ring *ring)
>>>> -{
>>>> -     return ring->entry_index < mux->ring_entry_size ?
>>>> -                     &mux->ring_entry[ring->entry_index] : NULL;
>>>> -}
>>>> -
>>>> -/* copy packages on sw ring range[begin, end) */ -static void
>>>> amdgpu_ring_mux_copy_pkt_from_sw_ring(struct amdgpu_ring_mux *mux,
>>>> -                                               struct amdgpu_ring *ring,
>>>> -                                               u64 s_start, u64 s_end)
>>>> -{
>>>> -     u64 start, end;
>>>> -     struct amdgpu_ring *real_ring = mux->real_ring;
>>>> -
>>>> -     start = s_start & ring->buf_mask;
>>>> -     end = s_end & ring->buf_mask;
>>>> -
>>>> -     if (start == end) {
>>>> -             DRM_ERROR("no more data copied from sw ring\n");
>>>> -             return;
>>>> -     }
>>>> -     if (start > end) {
>>>> -             amdgpu_ring_alloc(real_ring, (ring->ring_size >> 2) + end - start);
>>>> -             amdgpu_ring_write_multiple(real_ring, (void *)&ring->ring[start],
>>>> -                                        (ring->ring_size >> 2) - start);
>>>> -             amdgpu_ring_write_multiple(real_ring, (void *)&ring->ring[0], end);
>>>> -     } else {
>>>> -             amdgpu_ring_alloc(real_ring, end - start);
>>>> -             amdgpu_ring_write_multiple(real_ring, (void *)&ring->ring[start], end - start);
>>>> -     }
>>>> -}
>>>> -
>>>>     void amdgpu_ring_mux_set_wptr(struct amdgpu_ring_mux *mux, struct amdgpu_ring *ring, u64 wptr)
>>>>     {
>>>>         struct amdgpu_mux_entry *e;
>>>>
>>>> +     if (ring->hw_prio <= AMDGPU_RING_PRIO_DEFAULT)
>>>> +             amdgpu_mux_resubmit_chunks(mux, false);
>>>> +
>>>>         e = amdgpu_ring_mux_sw_entry(mux, ring);
>>>>         if (!e) {
>>>>                 DRM_ERROR("cannot find entry for sw ring\n");
>>>>                 return;
>>>>         }
>>>>
>>>> +     /* We could skip this set wptr as preemption in process. */
>>>> +     if (ring->hw_prio <= AMDGPU_RING_PRIO_DEFAULT && mux->pending_trailing_fence_signaled) {
>>>> +             DRM_ERROR("amdgpu_ring_mux_set_wptr skipped\n");
>>>> +             return;
>>>> +     }
>>>> +
>>>>         spin_lock(&mux->lock);
>>>>         e->sw_cptr = e->sw_wptr;
>>>> +     /* Update cptr if the package already copied in resubmit functions */
>>>> +     if (ring->hw_prio <= AMDGPU_RING_PRIO_DEFAULT && e->sw_cptr < mux->wptr_resubmit)
>>>> +             e->sw_cptr = mux->wptr_resubmit;
>>>>         e->sw_wptr = wptr;
>>>>         e->start_ptr_in_hw_ring = mux->real_ring->wptr;
>>>>
>>>> -     amdgpu_ring_mux_copy_pkt_from_sw_ring(mux, ring, e->sw_cptr, wptr);
>>>> -     e->end_ptr_in_hw_ring = mux->real_ring->wptr;
>>>> -     amdgpu_ring_commit(mux->real_ring);
>>>> -
>>>> +     /* Skip copying for the packages already resubmitted.*/
>>>> +     if (ring->hw_prio > AMDGPU_RING_PRIO_DEFAULT || mux->wptr_resubmit < wptr) {
>>>> +             amdgpu_ring_mux_copy_pkt_from_sw_ring(mux, ring, e->sw_cptr, wptr);
>>>> +             e->end_ptr_in_hw_ring = mux->real_ring->wptr;
>>>> +             amdgpu_ring_commit(mux->real_ring);
>>>> +     } else {
>>>> +             e->end_ptr_in_hw_ring = mux->real_ring->wptr;
>>>> +     }
>>>>         spin_unlock(&mux->lock);
>>>>     }
>>>>
>>>> @@ -181,3 +288,145 @@ u64 amdgpu_ring_mux_get_rptr(struct
>>>> amdgpu_ring_mux *mux, struct amdgpu_ring *ri
>>>>
>>>>         return e->sw_rptr;
>>>>     }
>>>> +
>>>> +void amdgpu_ring_mux_schedule_resubmit(struct amdgpu_ring_mux *mux) {
>>>> +     mod_timer(&mux->resubmit_timer, jiffies +
>>>> +AMDGPU_MUX_RESUBMIT_JIFFIES_TIMEOUT);
>>>> +}
>>>> +
>>>> +void amdgpu_ring_mux_start_ib(struct amdgpu_ring_mux *mux, struct
>>>> +amdgpu_ring *ring) {
>>>> +     struct amdgpu_mux_entry *e;
>>>> +     struct amdgpu_mux_chunk *chunk;
>>>> +
>>>> +     amdgpu_mux_resubmit_chunks(mux, false);
>>>> +
>>>> +     e = amdgpu_ring_mux_sw_entry(mux, ring);
>>>> +     if (!e) {
>>>> +             DRM_ERROR("cannot find entry!\n");
>>>> +             return;
>>>> +     }
>>>> +
>>>> +     chunk = kmem_cache_alloc(amdgpu_mux_chunk_slab, GFP_KERNEL);
>>>> +     if (!chunk) {
>>>> +             DRM_ERROR("alloc amdgpu_mux_chunk_slab failed\n");
>>>> +             return;
>>>> +     }
>>>> +
>>>> +     chunk->start = ring->wptr;
>>>> +     list_add_tail(&chunk->entry, &e->list); }
>>>> +
>>>> +static void scan_and_remove_signaled_chunk(struct amdgpu_ring_mux
>>>> +*mux, struct amdgpu_ring *ring) {
>>>> +     uint32_t last_seq, size = 0;
>>>> +     struct amdgpu_mux_entry *e;
>>>> +     struct amdgpu_mux_chunk *chunk, *tmp;
>>>> +
>>>> +     e = amdgpu_ring_mux_sw_entry(mux, ring);
>>>> +     if (!e) {
>>>> +             DRM_ERROR("cannot find entry!\n");
>>>> +             return;
>>>> +     }
>>>> +
>>>> +     last_seq = atomic_read(&ring->fence_drv.last_seq);
>>>> +
>>>> +     list_for_each_entry_safe(chunk, tmp, &e->list, entry) {
>>>> +             if (chunk->sync_seq <= last_seq) {
>>>> +                     list_del(&chunk->entry);
>>>> +                     kmem_cache_free(amdgpu_mux_chunk_slab, chunk);
>>>> +             } else {
>>>> +                     size++;
>>>> +             }
>>>> +     }
>>>> +}
>>>> +
>>>> +void amdgpu_ring_mux_end_ib(struct amdgpu_ring_mux *mux, struct
>>>> +amdgpu_ring *ring) {
>>>> +     struct amdgpu_mux_entry *e;
>>>> +     struct amdgpu_mux_chunk *chunk;
>>>> +
>>>> +     e = amdgpu_ring_mux_sw_entry(mux, ring);
>>>> +     if (!e) {
>>>> +             DRM_ERROR("cannot find entry!\n");
>>>> +             return;
>>>> +     }
>>>> +
>>>> +     chunk = list_last_entry(&e->list, struct amdgpu_mux_chunk, entry);
>>>> +     if (!chunk) {
>>>> +             DRM_ERROR("cannot find chunk!\n");
>>>> +             return;
>>>> +     }
>>>> +
>>>> +     chunk->end = ring->wptr;
>>>> +     chunk->sync_seq = READ_ONCE(ring->fence_drv.sync_seq);
>>>> +
>>>> +     scan_and_remove_signaled_chunk(mux, ring); }
>>>> +
>>>> +/* Trigger Mid-Command Buffer Preemption (MCBP) and find if we need
>>>> +to resubmit. */ int amdgpu_mcbp_trigger_preempt(struct
>>>> +amdgpu_ring_mux *mux) {
>>>> +     int r;
>>>> +
>>>> +     spin_lock(&mux->lock);
>>>> +     mux->pending_trailing_fence_signaled = true;
>>>> +     r = amdgpu_ring_preempt_ib(mux->real_ring);
>>>> +     spin_unlock(&mux->lock);
>>>> +     return r;
>>>> +}
>>>> +
>>>> +bool amdgpu_mcbp_handle_trailing_fence_irq(struct amdgpu_ring_mux
>>>> +*mux) {
>>>> +     struct amdgpu_mux_entry *e;
>>>> +     struct amdgpu_ring *ring = NULL;
>>>> +     int i;
>>>> +
>>>> +     if (!mux->pending_trailing_fence_signaled)
>>>> +             return false;
>>>> +
>>>> +     if (mux->real_ring->trail_seq != le32_to_cpu(*mux->real_ring->trail_fence_cpu_addr))
>>>> +             return false;
>>>> +
>>>> +     for (i = 0; i < mux->num_ring_entries; i++) {
>>>> +             e = &mux->ring_entry[i];
>>>> +             if (e->ring->hw_prio <= AMDGPU_RING_PRIO_DEFAULT) {
>>>> +                     ring = e->ring;
>>>> +                     break;
>>>> +             }
>>>> +     }
>>>> +
>>>> +     if (!ring) {
>>>> +             DRM_ERROR("cannot find low priority ring\n");
>>>> +             return false;
>>>> +     }
>>>> +
>>>> +     amdgpu_fence_process(ring);
>>>> +     if (amdgpu_fence_count_emitted(ring) > 0) {
>>>> +             mux->s_resubmit = true;
>>>> +             mux->seqno_to_resubmit = ring->fence_drv.sync_seq;
>>>> +             amdgpu_ring_mux_schedule_resubmit(mux);
>>>> +     }
>>>> +
>>>> +     mux->pending_trailing_fence_signaled = false;
>>>> +     return true;
>>>> +}
>>>> +
>>>> +/*scan on low prio rings to have unsignaled fence and high ring has
>>>> +no fence.*/ int amdgpu_mcbp_scan(struct amdgpu_ring_mux *mux) {
>>>> +     struct amdgpu_ring *ring;
>>>> +     int i, need_preempt;
>>>> +
>>>> +     need_preempt = 0;
>>>> +     for (i = 0; i < mux->num_ring_entries; i++) {
>>>> +             ring = mux->ring_entry[i].ring;
>>>> +             if (ring->hw_prio > AMDGPU_RING_PRIO_DEFAULT &&
>>>> +                     amdgpu_fence_count_emitted(ring) > 0)
>>>> +                     return 0;
>>>> +             if (ring->hw_prio <= AMDGPU_RING_PRIO_DEFAULT &&
>>>> +                     amdgpu_fence_count_emitted(ring) > 0)
>>>> +                     need_preempt = 1;
>>>> +     }
>>>> +     return need_preempt && !mux->s_resubmit; }
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.h
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.h
>>>> index 8c1691e11b1c..bf8f5ca61605 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.h
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring_mux.h
>>>> @@ -29,6 +29,7 @@
>>>>     #include "amdgpu_ring.h"
>>>>
>>>>     struct amdgpu_ring;
>>>> +
>>>>     /**
>>>>      * struct amdgpu_mux_entry - the entry recording software rings copying information.
>>>>      * @ring: the pointer to the software ring.
>>>> @@ -37,6 +38,7 @@ struct amdgpu_ring;
>>>>      * @sw_cptr: the position of the copy pointer in the sw ring.
>>>>      * @sw_rptr: the read pointer in software ring.
>>>>      * @sw_wptr: the write pointer in software ring.
>>>> + * @list: list head for amdgpu_mux_chunk
>>>>      */
>>>>     struct amdgpu_mux_entry {
>>>>         struct                  amdgpu_ring *ring;
>>>> @@ -45,6 +47,7 @@ struct amdgpu_mux_entry {
>>>>         u64                     sw_cptr;
>>>>         u64                     sw_rptr;
>>>>         u64                     sw_wptr;
>>>> +     struct list_head        list;
>>>>     };
>>>>
>>>>     struct amdgpu_ring_mux {
>>>> @@ -55,6 +58,26 @@ struct amdgpu_ring_mux {
>>>>         unsigned int            ring_entry_size;
>>>>         /*the lock for copy data from different software rings*/
>>>>         spinlock_t              lock;
>>>> +     bool                    s_resubmit;
>>>> +     uint32_t                seqno_to_resubmit;
>>>> +     u64                     wptr_resubmit;
>>>> +     struct timer_list       resubmit_timer;
>>>> +
>>>> +     bool                    pending_trailing_fence_signaled;
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct amdgpu_mux_chunk - save the location of indirect buffer's package on softare rings.
>>>> + * @entry: the list entry.
>>>> + * @sync_seq: the fence seqno related with the saved IB.
>>>> + * @start:- start location on the software ring.
>>>> + * @end:- end location on the software ring.
>>>> + */
>>>> +struct amdgpu_mux_chunk {
>>>> +     struct list_head        entry;
>>>> +     uint32_t                sync_seq;
>>>> +     u64                     start;
>>>> +     u64                     end;
>>>>     };
>>>>
>>>>     int amdgpu_ring_mux_init(struct amdgpu_ring_mux *mux, struct
>>>> amdgpu_ring *ring, @@ -65,4 +88,11 @@ void amdgpu_ring_mux_set_wptr(struct amdgpu_ring_mux *mux, struct amdgpu_ring *r
>>>>     u64 amdgpu_ring_mux_get_wptr(struct amdgpu_ring_mux *mux, struct amdgpu_ring *ring);
>>>>     u64 amdgpu_ring_mux_get_rptr(struct amdgpu_ring_mux *mux, struct
>>>> amdgpu_ring *ring);
>>>>
>>>> +void amdgpu_ring_mux_start_ib(struct amdgpu_ring_mux *mux, struct
>>>> +amdgpu_ring *ring); void amdgpu_ring_mux_end_ib(struct
>>>> +amdgpu_ring_mux *mux, struct amdgpu_ring *ring); void
>>>> +amdgpu_ring_mux_schedule_resubmit(struct amdgpu_ring_mux *mux);
>>>> +
>>>> +int amdgpu_mcbp_trigger_preempt(struct amdgpu_ring_mux *mux); int
>>>> +amdgpu_mcbp_scan(struct amdgpu_ring_mux *mux); bool
>>>> +amdgpu_mcbp_handle_trailing_fence_irq(struct amdgpu_ring_mux *mux);
>>>>     #endif
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_sw_ring.c
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_sw_ring.c
>>>> index 5ae12d6641ca..a3ec7bdf72a6 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_sw_ring.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_sw_ring.c
>>>> @@ -59,3 +59,29 @@ void amdgpu_sw_ring_insert_nop(struct amdgpu_ring *ring, uint32_t count)
>>>>     {
>>>>         WARN_ON(!ring->is_sw_ring);
>>>>     }
>>>> +
>>>> +void amdgpu_sw_ring_ib_begin(struct amdgpu_ring *ring) {
>>>> +     struct amdgpu_device *adev = ring->adev;
>>>> +     struct amdgpu_ring_mux *mux = &adev->gfx.muxer;
>>>> +
>>>> +     WARN_ON(!ring->is_sw_ring);
>>>> +     if (ring->hw_prio > AMDGPU_RING_PRIO_DEFAULT) {
>>>> +             if (amdgpu_mcbp_scan(mux) > 0)
>>>> +                     amdgpu_mcbp_trigger_preempt(mux);
>>>> +             return;
>>>> +     }
>>>> +
>>>> +     amdgpu_ring_mux_start_ib(mux, ring); }
>>>> +
>>>> +void amdgpu_sw_ring_ib_end(struct amdgpu_ring *ring) {
>>>> +     struct amdgpu_device *adev = ring->adev;
>>>> +     struct amdgpu_ring_mux *mux = &adev->gfx.muxer;
>>>> +
>>>> +     WARN_ON(!ring->is_sw_ring);
>>>> +     if (ring->hw_prio > AMDGPU_RING_PRIO_DEFAULT)
>>>> +             return;
>>>> +     amdgpu_ring_mux_end_ib(mux, ring); }
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>> index 9596c22fded6..b7e94553f4fb 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>>> @@ -601,6 +601,7 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
>>>>         if (!vm_flush_needed && !gds_switch_needed && !need_pipe_sync)
>>>>                 return 0;
>>>>
>>>> +     amdgpu_ring_ib_begin(ring);
>>>>         if (ring->funcs->init_cond_exec)
>>>>                 patch_offset = amdgpu_ring_init_cond_exec(ring);
>>>>
>>>> @@ -661,6 +662,7 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
>>>>                 amdgpu_ring_emit_switch_buffer(ring);
>>>>                 amdgpu_ring_emit_switch_buffer(ring);
>>>>         }
>>>> +     amdgpu_ring_ib_end(ring);
>>>>         return 0;
>>>>     }
>>>>
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
>>>> b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
>>>> index 669532f658da..1620300f0dde 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
>>>> @@ -5619,7 +5619,7 @@ static int gfx_v9_0_ring_preempt_ib(struct amdgpu_ring *ring)
>>>>         ring->trail_seq += 1;
>>>>         amdgpu_ring_alloc(ring, 13);
>>>>         gfx_v9_0_ring_emit_fence(ring, ring->trail_fence_gpu_addr,
>>>> -                              ring->trail_seq, AMDGPU_FENCE_FLAG_EXEC);
>>>> +                              ring->trail_seq, AMDGPU_FENCE_FLAG_EXEC
>>>> + | AMDGPU_FENCE_FLAG_INT);
>>>>         /*reset the CP_VMID_PREEMPT after trailing fence*/
>>>>         amdgpu_ring_emit_wreg(ring,
>>>>                               SOC15_REG_OFFSET(GC, 0,
>>>> mmCP_VMID_PREEMPT), @@ -6045,9 +6045,11 @@ static int
>>>> gfx_v9_0_eop_irq(struct amdgpu_device *adev,
>>>>
>>>>         switch (me_id) {
>>>>         case 0:
>>>> -             /* Fence signals are handled on the software rings*/
>>>> -             for (i = 0; i < GFX9_NUM_SW_GFX_RINGS; i++)
>>>> -                     amdgpu_fence_process(&adev->gfx.sw_gfx_ring[i]);
>>>> +             if (!amdgpu_mcbp_handle_trailing_fence_irq(&adev->gfx.muxer)) {
>>>> +                     /* Fence signals are handled on the software rings*/
>>>> +                     for (i = 0; i < GFX9_NUM_SW_GFX_RINGS; i++)
>>>> +                             amdgpu_fence_process(&adev->gfx.sw_gfx_ring[i]);
>>>> +             }
>>>>                 break;
>>>>         case 1:
>>>>         case 2:



More information about the amd-gfx mailing list