[Intel-gfx] [PATCH 1/1] drm/i915/guc: Fix GuC error capture sizing estimation and reporting

[John Harrison]

On 10/3/2022 14:10, Teres Alexis, Alan Previn wrote:
> On Mon, 2022-10-03 at 12:47 -0700, Harrison, John C wrote:
>> On 10/3/2022 11:28, Teres Alexis, Alan Previn wrote:
>>> On Fri, 2022-09-30 at 15:35 -0700, Harrison, John C wrote:
>>>> On 9/30/2022 14:08, Teres Alexis, Alan Previn wrote:
>>>>> I disagree because its unlikely that all engines can reset all at once (we probably have bigger problems at the at
>>>>> point) and if they all occurred within the same G2H handler scheduled worker run, our current gpu_coredump framework
>>>>> would just discard the ones after the first one and so it wouldnt even matter if we did catch it.
>>>> So min_size is not actually the minimal size for a meaningful capture?
>>>> So what is? And remember that for compute class engines, there is
>>>> dependent engine reset. So a reset of CCS2 also means a reset of RCS,
>>>> CCS0, CCS1 and CCS3. So having at least four engines per capture is not
>>>> unreasonable.
>>>>
>>> Alan: min_size is a meaningful size for the worst case scenario of collecting the guc-report. However due to gpu-core-
>>> dump, its not meaningful in terms of reporting that information out to the user. Thats not a limitation of this
>>> subsystem.
>> I'm not following what you mean about gpu-core-dump. The point of the
>> warning is to let the user know that the interface they want to use
>> (error capture via sysfs) might not be complete in the majority of
>> cases. If there is some catastrophic worst case scenario that overflows
>> but 99% of instances are fine then that's what the drm_notice about
>> 'spare_size' is for. But if the 99% of instances of a capture are going
>> to overflow then that is what the drm_warn about 'min_size' should be for.
>>
>>>> It seems pointless to go through a lot of effort to calculate the
>>>> minimum and recommend sizes only to basically ignore them by just
>>>> whispering very, very quietly that there might be a problem.
>>>>
>>> Alan: I already responded that i will re-rev as per your recommendation and switch back to drm_notice.
>>>
>>>> It also
>>>> seems pointless to complain about a minimum size that actually isn't the
>>>> minimum size. That's sort of worse - now you are telling the user there
>>>> is a problem when really there isn't.
>>>>
>>> Alan: the min size is accurate - but whether we report to the use about them possibly facing a problem is where it gets
>>> a bit unclear because of the limitation in the gpu-core-dump framework. We could drop the message completely if you like
>>> - but then we'd have to remember to re-add it if we fix gpu-core-dump in future. For now, as i mentioned in the last
>> What limitation in the 'gpu-core-dump framework'? What is the fix required?
>>
> Unless it has changed since i last looked at it, gpu_coredump framework will store the first error-state captured and
> wait until the user extracts it via sysfs. Any other error-state capture that gets created before that prior one was
> cleared by the user gets dropped. I'm not sure if that limitation needs to be "fixed". I am not sure of its history -
> maybe it was designed that way for a reason. ATM I dont have the bandwidth to chase that history down.
That is by design. Generally speaking, you want to see the first error 
that occurred. If there were a bunch back to back then likely the first 
one is the real problem and the rest are either symptoms or repeat 
offences of the first.

Also, in the case where a user doesn't know what an error capture is but 
they are running (or developing) some app that keeps dying and produces 
core dumps, you can't just swallow all system memory in a linked list of 
error captures that won't ever be read out and freed up.

Hence the design is to save the first and hold on to it until read out. 
And if any further hangs happen before the current one is cleared, they 
are discarded as uninteresting if not plain old spam.

>
>>> reply, i have already changed it back to "notice" as per your last comment. Perhaps you should have looked at rev2 which
>>> was posted before your responses above. As mentioned in last reply, i disagreed but i am committing to your request
>>> which was fixed in rev2 as per your request.
>> The point of a code review is not "I say X so you must do X, immediately
>> post a new revision now". I asked some questions. I stated my opinion
>> about what the end user should see. If you think your implementation
>> already matches that or you disagree because you think I am basing my
>> comments on incorrect information, or even just that you disagree with
>> my reasoning, then you should not blindly post a new revision saying
>> "here are your changes, I don't like it because Y but just r-b it and
>> I'll merge". You should reply to the comments with your thoughts and
>> suggestions. Maybe the reviewer is wrong!
>>
> I think this comes down to preference as there is no clear rule about what is right vs what is wrong in this case:
>
> 	- we have a possible issue but its a corner case when all engine instances suffer a reset all at once.
> 	- in the event of that corner case occurring, the end-user could go back and increase the guc-log-capture-
> region size and thus the warning will go away and GuC will have space to report the error-state dump for all engine
> instances if they got reset all at once (within the same G2H IRQ event).
> 	- assuming the issue is reproducible, the user, now happy that the warning is resolved, reruns the workload.
> The
> user dumps the first error-state-capture via sysfs which only contains info for the first engine that was reset (as it
> appeared in the G2H-CTB). the user inspects the same sysfs and none of the other dmesg-reported engine-hang error-state-
> dumps seem to be made available in the sysfs after clearing that first one.
>
> so the preference here becomes "should we warn the user about this possible event of running out of guc-error-state
> capture region when resolving it still doesn't allow the user to get it anyway"... OR "should we just make a quiet debug
> message about it since an i915 developer knows that the gpu_coredump framework design can only ever store 1 error-state-
> capture-dump and throws the rest away until its cleared by the user via sysfs".
>
> My preference is the latter. However, just to be clear, with the fixed calculation, we'll probably never see the
> warning(or-debug) message with the HW that we have today and the foreseeable future.

The warning isn't meant to be for i915 developers who understand bizarre 
internal workings of the driver. The warning is meant to be for end 
users who might not be able to debug their applications.

Given that the error capture interface is fundamentally only really 
concerned about obtaining a single valid capture at a time. I would 
argue that supporting a dozen back-to-back captures is no more than a 
nice-to-have convenience for a very rare situation (such as the user has 
explicitly modified their kernel to save all the captures in a list). 
And that back-to-back scenario is the 'spare_size', not 'min_size' isn't 
it? So there is no need for a warning about that. A notice is fine but 
not a warning.

The thing that really, really, really matters to an end user is that the 
first capture of a session is complete and useful. So my view is that 
any warning message should be focused on that. Can we get one complete 
error capture for one complete set of engines? If not, that is something 
we should warn about. And that is what 'min_size' should be.

Note that 'one complete set of engines' really only means five on 
current hardware - RCS + CCS0-3 is the most that can reset at one time 
(due to dependent engine resets) given that GuC managed resets are 
context based and a context can only run on one engine. Although I think 
it is technically possible for more engines to be involved but that 
would also count as hard to hit a corner case. And given the small size 
involved, it might be simplest just to base min_size on all engines anyway.



>>>> IMHO, the min_size check should be meaningful and should be visible to
>>>> the user if it fails.
>>>>
>>>> Also, are we still hitting the minimum size failure message? Now that
>>>> the calculation has been fixed, what sizes does it come up with for min
>>>> and spare? Are they within the allocation now or not?
>>>>
>>> Yes - we are within the allocation with this patch (the fix of removing the redundant register-struct-size
>>> multiplication brought the number down significantly).
>> Bringing in comment from other thread:
>>   > Some context: with the calculation fix we are allocating 4MB but we
>> only need 78k as min-est.
>>
>> Wow! That is quite a big smaller. And much more plausible! So if
>> min_size is 78KB, what is spare_size now?
>>
> Around 230K. (the min_est was exactly 78504 bytes or 76.664Kb - that was based on a DG2 - i think it was a 384 - cant
> recall for sure)
Might be worth checking on PVC. That could be getting close to 1MB with 
all the extra blitter engines.

>
>> Sounds like we can drop it down to a 1MB allocation.
>>
> Okay - will do a 3rd rev with a 1MB change.
Don't start writing anything yet. Let's get to a final decision on the 
plan first.

>
>> And even if
>> min_size is not the absolute minimum but quite a bit more worst case
>> kind of size, it still seems reasonable to keep it as a warning rather
>> than a notice. Given that it is so much smaller than the allocation
>> size, if it does somehow overflow on some platform/configuration then
>> that sounds like something we should know about because it likely means
>> something is broken.
> Okay - this works fine too since we probably will likely never trigger that message until we have significantly more
> engine instances than todays hw.
See above about PVC.

John.