<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 4, 2016 at 1:24 PM, Daniel Vetter <span dir="ltr"><<a href="mailto:daniel@ffwll.ch" target="_blank">daniel@ffwll.ch</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class="h5">On Wed, May 04, 2016 at 10:49:53AM +0100, Robert Bragg wrote:<br>
> On Wed, May 4, 2016 at 10:09 AM, Martin Peres <<a href="mailto:martin.peres@linux.intel.com">martin.peres@linux.intel.com</a>><br>
> wrote:<br>
><br>
> > On 03/05/16 23:03, Robert Bragg wrote:<br>
> ><br>
> >><br>
> >><br>
> >> On Tue, May 3, 2016 at 8:34 PM, Robert Bragg <<a href="mailto:robert@sixbynine.org">robert@sixbynine.org</a><br>
> >> <mailto:<a href="mailto:robert@sixbynine.org">robert@sixbynine.org</a>>> wrote:<br>
> >><br>
> >> Sorry for the delay replying to this, I missed it.<br>
> >><br>
> >> On Sat, Apr 23, 2016 at 11:34 AM, Martin Peres <<a href="mailto:martin.peres@free.fr">martin.peres@free.fr</a><br>
> >> <mailto:<a href="mailto:martin.peres@free.fr">martin.peres@free.fr</a>>> wrote:<br>
> >><br>
> >> On 20/04/16 17:23, Robert Bragg wrote:<br>
> >><br>
> >> Gen graphics hardware can be set up to periodically write<br>
> >> snapshots of<br>
> >> performance counters into a circular buffer via its<br>
> >> Observation<br>
> >> Architecture and this patch exposes that capability to<br>
> >> userspace via the<br>
> >> i915 perf interface.<br>
> >><br>
> >> Cc: Chris Wilson <<a href="mailto:chris@chris-wilson.co.uk">chris@chris-wilson.co.uk</a><br>
> >> <mailto:<a href="mailto:chris@chris-wilson.co.uk">chris@chris-wilson.co.uk</a>>><br>
> >> Signed-off-by: Robert Bragg <<a href="mailto:robert@sixbynine.org">robert@sixbynine.org</a><br>
> >> <mailto:<a href="mailto:robert@sixbynine.org">robert@sixbynine.org</a>>><br>
> >> Signed-off-by: Zhenyu Wang <<a href="mailto:zhenyuw@linux.intel.com">zhenyuw@linux.intel.com</a><br>
> >> <mailto:<a href="mailto:zhenyuw@linux.intel.com">zhenyuw@linux.intel.com</a>>><br>
> >><br>
> >> ---<br>
> >> drivers/gpu/drm/i915/i915_drv.h | 56 +-<br>
> >> drivers/gpu/drm/i915/i915_gem_context.c | 24 +-<br>
> >> drivers/gpu/drm/i915/i915_perf.c | 940<br>
> >> +++++++++++++++++++++++++++++++-<br>
> >> drivers/gpu/drm/i915/i915_reg.h | 338 ++++++++++++<br>
> >> include/uapi/drm/i915_drm.h | 70 ++-<br>
> >> 5 files changed, 1408 insertions(+), 20 deletions(-)<br>
> >><br>
> >> +<br>
> >> +<br>
> >> + /* It takes a fairly long time for a new MUX<br>
> >> configuration to<br>
> >> + * be be applied after these register writes. This<br>
> >> delay<br>
> >> + * duration was derived empirically based on the<br>
> >> render_basic<br>
> >> + * config but hopefully it covers the maximum<br>
> >> configuration<br>
> >> + * latency...<br>
> >> + */<br>
> >> + mdelay(100);<br>
> >><br>
> >><br>
> >> With such a HW and SW design, how can we ever expose hope to get<br>
> >> any<br>
> >> kind of performance when we are trying to monitor different<br>
> >> metrics on each<br>
> >> draw call? This may be acceptable for system monitoring, but it<br>
> >> is problematic<br>
> >> for the GL extensions :s<br>
> >><br>
> >><br>
> >> Since it seems like we are going for a perf API, it means that<br>
> >> for every change<br>
> >> of metrics, we need to flush the commands, wait for the GPU to<br>
> >> be done, then<br>
> >> program the new set of metrics via an IOCTL, wait 100 ms, and<br>
> >> then we may<br>
> >> resume rendering ... until the next change. We are talking about<br>
> >> a latency of<br>
> >> 6-7 frames at 60 Hz here... this is non-negligeable...<br>
> >><br>
> >><br>
> >> I understand that we have a ton of counters and we may hide<br>
> >> latency by not<br>
> >> allowing using more than half of the counters for every draw<br>
> >> call or frame, but<br>
> >> even then, this 100ms delay is killing this approach altogether.<br>
> >><br>
> >><br>
> >><br>
> >><br>
> >> So revisiting this to double check how things fail with my latest<br>
> >> driver/tests without the delay, I apparently can't reproduce test<br>
> >> failures without the delay any more...<br>
> >><br>
> >> I think the explanation is that since first adding the delay to the<br>
> >> driver I also made the the driver a bit more careful to not forward<br>
> >> spurious reports that look invalid due to a zeroed report id field, and<br>
> >> that mechanism keeps the unit tests happy, even though there are still<br>
> >> some number of invalid reports generated if we don't wait.<br>
> >><br>
> >> One problem with simply having no delay is that the driver prints an<br>
> >> error if it sees an invalid reports so I get a lot of 'Skipping<br>
> >> spurious, invalid OA report' dmesg spam. Also this was intended more as<br>
> >> a last resort mechanism, and I wouldn't feel too happy about squashing<br>
> >> the error message and potentially sweeping other error cases under the<br>
> >> carpet.<br>
> >><br>
> >> Experimenting to see if the delay can at least be reduced, I brought the<br>
> >> delay up in millisecond increments and found that although I still see a<br>
> >> lot of spurious reports only waiting 1 or 5 milliseconds, at 10<br>
> >> milliseconds its reduced quite a bit and at 15 milliseconds I don't seem<br>
> >> to have any errors.<br>
> >><br>
> >> 15 milliseconds is still a long time, but at least not as long as 100.<br>
> >><br>
> ><br>
> > OK, so the issue does not come from the HW after all, great!<br>
> ><br>
><br>
> Erm, I'm not sure that's a conclusion we can make here...<br>
><br>
> The upshot here was really just reducing the delay from 100ms to 15ms.<br>
> Previously I arrived at a workable delay by jumping the delay in orders of<br>
> magnitude with 10ms failing, 100ms passing and I didn't try and refine it<br>
> further. Here I've looked at delays between 10 and 100ms.<br>
><br>
> The other thing is observing that because the kernel is checking for<br>
> invalid reports (generated by the hardware) before forwarding to userspace<br>
> the lack of a delay no longer triggers i-g-t failures because the invalid<br>
> data won't reach i-g-t any more - though the invalid reports are still a<br>
> thing to avoid.<br>
><br>
><br>
> > Now, my main question is, why are spurious events generated when changing<br>
> > the MUX's value? I can understand that we would need to ignore the reading<br>
> > that came right after the change, but other than this, I am a bit at a<br>
> > loss.<br>
> ><br>
><br>
> The MUX selects 16 signals that the OA unit can turn into 16 separate<br>
> counters by basically counting the signal changes. (there's some fancy<br>
> fixed function logic that can affect this but that's the general idea).<br>
><br>
> If the MUX is in the middle of being re-programmed then some subset of<br>
> those 16 signals are for who knows what.<br>
><br>
> After programming the MUX we will go on to configure the OA unit and the<br>
> tests will enable periodic sampling which (if we have no delay) will sample<br>
> the OA counters that are currently being fed by undefined signals.<br>
><br>
> So as far as that goes it makes sense to me to expect bad data if we don't<br>
> wait for the MUX config to land properly. Something I don't really know is<br>
> how come we're seemingly lucky to have the reports be cleanly invalid with<br>
> a zero report-id, instead of just having junk data that would be harder to<br>
> recognise.<br>
<br>
</div></div>Yeah this mdelay story sounds realy scary. Few random comments:<br>
- msleep instead of mdelay please<br></blockquote><div> </div><div>yup this was a mistake I'd forgotten to fix in this series, but is fixed in the last series I sent after chris noticed too.<br><br></div><div>actually in my latest (yesterday after experimenting further with the delay requirements) I'm using usleep_range for a delay between 15 and 20 milliseconds which seems to be enough.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- no dmesg noise above debug level for stuff that we know can happen -<br>
dmesg noise counts as igt failures<br></blockquote><div><br></div><div>okey. I don't think I have anything above debug level, unless things are going badly wrong.<br><br>Just double checking though has made me think twice about a WARN_ON in gen7_oa_read for !oa_buffer_addr, which would be a bad situation but should either be removed (never expected), be a BUG_ON (since the code would deref NULL anyway) or more gracefully return an error if seen.<br><br>I currently have some DRM_DRIVER_DEBUG errors for cases where userspace messes up what properties it gives to open a stream - hopefully that sound ok? I've found it quite helpful to have a readable error for otherwise vague EINVAL type errors.<br></div><div><br></div><div>I have a debug message I print if we see an invalid HW report, which *shouldn't* happen but can happen (e.g. if the MUX delay or tail margin aren't well tuned) and it's helpful to have the feedback, in case we end up in a situation where we see this kind of message too frequently which might indicate an issue to investigate.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- reading 0 sounds more like bad synchronization. </blockquote><br><div>I suppose I haven't thoroughly considered if we should return zero in any case - normally that would imply EOF so we get to choose what that implies here. I don't think the driver should ever return 0 from read() currently.<br><br></div><div>A few concious choices re: read() return values have been:<br><br></div><div>- never ever return partial records (or rather even if a partial record were literally copied into the userspace buffer, but an error were hit in the middle of copying a full sample then that record wouldn't be accounted for in the byte count returned.)<br><br></div><div>- Don't throw away records successfully copied, due to a later error. This simplifies error handling paths internally and reporting EAGAIN/ENOSPC/EFAULT errors and means data isn't lost. The precedence for what we return is 1) did we successfully copy some reports? report bytes copied for complete records. 2) did we encounter an error? report that if so. 3) return -EAGAIN. (though for a blocking fd this will be handled internally).<br><br></div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Have you tried quiescing </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">the entire gpu (to make sure nothing is happen) and disabling OA, then<br>
updating, and then restarting? At least on a very quick look I didn't<br>
spot that. Random delays freak me out a bit, but wouldn't be surprised<br>
if really needed. <br></blockquote><div><br></div><div>Experimenting yesterday, it seems I can at least reduce the delay to around 15ms (granted that's still pretty huge), and it's also workable to have userspace sleep for this time (despite the mdelay I originally went with)<br></div><div><br></div><div>Haven't tried this, but yeah could be interesting to experiment if the MUX config lands faster in different situation such as when the HW is idle.<br><br></div><div>Thanks,<br></div><div>- Robert<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Cheers, Daniel<br>
<span class=""><font color="#888888">--<br>
Daniel Vetter<br>
Software Engineer, Intel Corporation<br>
<a href="http://blog.ffwll.ch" rel="noreferrer" target="_blank">http://blog.ffwll.ch</a><br>
</font></span></blockquote></div></div><br></div></div>