[PATCH 4/7] drm/msm/A6xx: Implement preemption for A7XX targets

[Antonino Maniscalco]

On 8/27/24 11:07 PM, Rob Clark wrote:
> On Tue, Aug 27, 2024 at 1:25 PM Antonino Maniscalco
> <antomani103 at gmail.com> wrote:
>>
>> On 8/27/24 9:48 PM, Akhil P Oommen wrote:
>>> On Fri, Aug 23, 2024 at 10:23:48AM +0100, Connor Abbott wrote:
>>>> On Fri, Aug 23, 2024 at 10:21 AM Connor Abbott <cwabbott0 at gmail.com> wrote:
>>>>>
>>>>> On Thu, Aug 22, 2024 at 9:06 PM Akhil P Oommen <quic_akhilpo at quicinc.com> wrote:
>>>>>>
>>>>>> On Wed, Aug 21, 2024 at 05:02:56PM +0100, Connor Abbott wrote:
>>>>>>> On Mon, Aug 19, 2024 at 9:09 PM Akhil P Oommen <quic_akhilpo at quicinc.com> wrote:
>>>>>>>>
>>>>>>>> On Thu, Aug 15, 2024 at 08:26:14PM +0200, Antonino Maniscalco wrote:
>>>>>>>>> This patch implements preemption feature for A6xx targets, this allows
>>>>>>>>> the GPU to switch to a higher priority ringbuffer if one is ready. A6XX
>>>>>>>>> hardware as such supports multiple levels of preemption granularities,
>>>>>>>>> ranging from coarse grained(ringbuffer level) to a more fine grained
>>>>>>>>> such as draw-call level or a bin boundary level preemption. This patch
>>>>>>>>> enables the basic preemption level, with more fine grained preemption
>>>>>>>>> support to follow.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Sharat Masetty <smasetty at codeaurora.org>
>>>>>>>>> Signed-off-by: Antonino Maniscalco <antomani103 at gmail.com>
>>>>>>>>> ---
>>>>>>>>
>>>>>>>> No postamble packets which resets perfcounters? It is necessary. Also, I
>>>>>>>> think we should disable preemption during profiling like we disable slumber.
>>>>>>>>
>>>>>>>> -Akhil.
>>>>>>>
>>>>>>> I don't see anything in kgsl which disables preemption during
>>>>>>> profiling. It disables resetting perfcounters when doing system-wide
>>>>>>> profiling, like freedreno, and in that case I assume preempting is
>>>>>>> fine because the system profiler has a complete view of everything and
>>>>>>> should "see" preemptions through the traces. For something like
>>>>>>> VK_KHR_performance_query I suppose we'd want to disable preemption
>>>>>>> because we disable saving/restoring perf counters, but that has to
>>>>>>> happen in userspace because the kernel doesn't know what userspace
>>>>>>> does.
>>>>>>>
>>>>>>
>>>>>> KGSL does some sort of arbitration of perfcounter configurations and
>>>>>> adds the select/enablement reg configuration as part of dynamic
>>>>>> power up register list which we are not doing here. Is this something
>>>>>> you are taking care of from userspace via preamble?
>>>>>>
>>>>>> -Akhil
>>>>>
>>>>> I don't think we have to take care of that in userspace, because Mesa
>>>>> will always configure the counter registers before reading them in the
>>>>> same submission, and if it gets preempted in the meantime then we're
>>>>> toast anyways (due to not saving/restoring perf counters). kgsl sets
>>>>> them from userspace, which is why it has to do something to set them
>>>>
>>>> Sorry, should be "kgsl sets them from the kernel".
>>>>
>>>>> after IFPC slumber or a context switch when the HW state is gone.
>>>>> Also, because the upstream approach doesn't play nicely with system
>>>>> profilers like perfetto, VK_KHR_performance_query is hidden by default
>>>>> behind a debug flag in turnip. So there's already an element of "this
>>>>> is unsupported, you have to know what you're doing to use it."
>>>
>>> But when you have composition on GPU enabled, there will be very frequent
>>> preemption. And I don't know how usable profiling tools will be in that
>>> case unless you disable preemption with a Mesa debug flag. But for that
>>> to work, all existing submitqueues should be destroyed and recreated.
>>>
>>> So I was thinking that we can use the sysprof propertry to force L0
>>> preemption from kernel.
>>>
>>> -Akhil.
>>>
>>
>> Right but when using a system profiler I imagined the expectation would
>> be to be able to understand how applications and compositor interact. An
>> use case could be measuring latency and understanding what contributes
>> to it. That is actually the main reason I added traces for preemption.
>> Disabling preemption would make it less useful for this type of
>> analysis. Did you have an use case in mind for a system profiler that
>> would benefit from disabling preemption and that is not covered by
>> VK_KHR_performance_query (or equivalent GL ext)?
> 
> I would think that we want to generate an event, with GPU timestamp
> (ie. RB_DONE) and which ring we are switching to, so that perfetto/etc
> could display multiple GPU timelines and where the switch from one to
> the other happens.
> 
> I'm a bit curious how this is handled on android, with AGI/etc.. I
> don't see any support in perfetto for this.
> 
> BR,
> -R
> 
>> Best regards,
>> --
>> Antonino Maniscalco <antomani103 at gmail.com>
>>

Looking at KGSL they seem to use ftrace and I don't see it doing 
anything to get a timestamp from some GPU timer, really not sure how 
that would be put in a gpu timeline.

Best regards,
-- 
Antonino Maniscalco <antomani103 at gmail.com>