<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple"> <div class="WordSection1"> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></a></p> <p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> sourab gupta [mailto:sourabgupta@gmail.com] <br> <b>Sent:</b> Wednesday, August 2, 2017 8:17 AM<br> <b>To:</b> Landwerlin, Lionel G <lionel.g.landwerlin@intel.com><br> <b>Cc:</b> Kamble, Sagar A <sagar.a.kamble@intel.com>; intel-gfx@lists.freedesktop.org; Sourab Gupta <sourab.gupta@intel.com><br> <b>Subject:</b> Re: [Intel-gfx] [PATCH 03/12] drm/i915: Framework for capturing command stream based OA reports and ctx id info.<o:p></o:p></span></p> <p class="MsoNormal"><o:p> </o:p></p> <div> <p class="MsoNormal"><o:p> </o:p></p> <div> <p class="MsoNormal"><o:p> </o:p></p> <div> <p class="MsoNormal">On Wed, Aug 2, 2017 at 2:28 AM, Lionel Landwerlin <<a href="mailto:lionel.g.landwerlin@intel.com" target="_blank">lionel.g.landwerlin@intel.com</a>> wrote:<o:p></o:p></p> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"> <div> <div> <div> <div> <p class="MsoNormal">On 01/08/17 19:05, sourab gupta wrote:<o:p></o:p></p> </div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <div> <p class="MsoNormal"><o:p> </o:p></p> <div> <p class="MsoNormal"><o:p> </o:p></p> <div> <p class="MsoNormal">On Tue, Aug 1, 2017 at 2:59 PM, Kamble, Sagar A <<a href="mailto:sagar.a.kamble@intel.com" target="_blank">sagar.a.kamble@intel.com</a>> wrote:<o:p></o:p></p> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"> <div> <div> <p class="MsoNormal" style="margin-bottom:12.0pt"><br> <br> -----Original Message-----<br> From: Landwerlin, Lionel G<br> Sent: Monday, July 31, 2017 9:16 PM<br> To: Kamble, Sagar A <<a href="mailto:sagar.a.kamble@intel.com" target="_blank">sagar.a.kamble@intel.com</a>>; <a href="mailto:intel-gfx@lists.freedesktop.org" target="_blank">intel-gfx@lists.freedesktop.org</a><br> Cc: Sourab Gupta <<a href="mailto:sourab.gupta@intel.com" target="_blank">sourab.gupta@intel.com</a>><br> Subject: Re: [Intel-gfx] [PATCH 03/12] drm/i915: Framework for capturing command stream based OA reports and ctx id info.<br> <br> On 31/07/17 08:59, Sagar Arun Kamble wrote:<br> > From: Sourab Gupta <<a href="mailto:sourab.gupta@intel.com" target="_blank">sourab.gupta@intel.com</a>><br> ><br> > This patch introduces a framework to capture OA counter reports associated<br> > with Render command stream. We can then associate the reports captured<br> > through this mechanism with their corresponding context id's. This can be<br> > further extended to associate any other metadata information with the<br> > corresponding samples (since the association with Render command stream<br> > gives us the ability to capture these information while inserting the<br> > corresponding capture commands into the command stream).<br> ><br> > The OA reports generated in this way are associated with a corresponding<br> > workload, and thus can be used the delimit the workload (i.e. sample the<br> > counters at the workload boundaries), within an ongoing stream of periodic<br> > counter snapshots.<br> ><br> > There may be usecases wherein we need more than periodic OA capture mode<br> > which is supported currently. This mode is primarily used for two usecases:<br> > - Ability to capture system wide metrics, alongwith the ability to map<br> > the reports back to individual contexts (particularly for HSW).<br> > - Ability to inject tags for work, into the reports. This provides<br> > visibility into the multiple stages of work within single context.<br> ><br> > The userspace will be able to distinguish between the periodic and CS based<br> > OA reports by the virtue of source_info sample field.<br> ><br> > The command MI_REPORT_PERF_COUNT can be used to capture snapshots of OA<br> > counters, and is inserted at BB boundaries.<br> > The data thus captured will be stored in a separate buffer, which will<br> > be different from the buffer used otherwise for periodic OA capture mode.<br> > The metadata information pertaining to snapshot is maintained in a list,<br> > which also has offsets into the gem buffer object per captured snapshot.<br> > In order to track whether the gpu has completed processing the node,<br> > a field pertaining to corresponding gem request is added, which is tracked<br> > for completion of the command.<br> ><br> > Both periodic and CS based reports are associated with a single stream<br> > (corresponding to render engine), and it is expected to have the samples<br> > in the sequential order according to their timestamps. Now, since these<br> > reports are collected in separate buffers, these are merge sorted at the<br> > time of forwarding to userspace during the read call.<br> ><br> > v2: Aligning with the non-perf interface (custom drm ioctl based). Also,<br> > few related patches are squashed together for better readability<br> ><br> > v3: Updated perf sample capture emit hook name. Reserving space upfront<br> > in the ring for emitting sample capture commands and using<br> > req->fence.seqno for tracking samples. Added SRCU protection for streams.<br> > Changed the stream last_request tracking to resv object. (Chris)<br> > Updated perf.sample_lock spin_lock usage to avoid softlockups. Moved<br> > stream to global per-engine structure. (Sagar)<br> > Update unpin and put in the free routines to i915_vma_unpin_and_release.<br> > Making use of perf stream cs_buffer vma resv instead of separate resv obj.<br> > Pruned perf stream vma resv during gem_idle. (Chris)<br> > Changed payload field ctx_id to u64 to keep all sample data aligned at 8<br> > bytes. (Lionel)<br> > stall/flush prior to sample capture is not added. Do we need to give this<br> > control to user to select whether to stall/flush at each sample?<br> ><br> > Signed-off-by: Sourab Gupta <<a href="mailto:sourab.gupta@intel.com" target="_blank">sourab.gupta@intel.com</a>><br> > Signed-off-by: Robert Bragg <<a href="mailto:robert@sixbynine.org" target="_blank">robert@sixbynine.org</a>><br> > Signed-off-by: Sagar Arun Kamble <<a href="mailto:sagar.a.kamble@intel.com" target="_blank">sagar.a.kamble@intel.com</a>><br> > ---<br> > drivers/gpu/drm/i915/i915_drv.h | 101 ++-<br> > drivers/gpu/drm/i915/i915_gem.c | 1 +<br> > drivers/gpu/drm/i915/i915_gem_execbuffer.c | 8 +<br> > drivers/gpu/drm/i915/i915_perf.c | 1185 ++++++++++++++++++++++------<br> > drivers/gpu/drm/i915/intel_engine_cs.c | 4 +<br> > drivers/gpu/drm/i915/intel_ringbuffer.c | 2 +<br> > drivers/gpu/drm/i915/intel_ringbuffer.h | 5 +<br> > include/uapi/drm/i915_drm.h | 15 +<br> > 8 files changed, 1073 insertions(+), 248 deletions(-)<br> ><br> > diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h<br> > index 2c7456f..8b1cecf 100644<br> > --- a/drivers/gpu/drm/i915/i915_drv.h<br> > +++ b/drivers/gpu/drm/i915/i915_drv.h<br> > @@ -1985,6 +1985,24 @@ struct i915_perf_stream_ops {<br> > * The stream will always be disabled before this is called.<br> > */<br> > void (*destroy)(struct i915_perf_stream *stream);<br> > +<br> > + /*<br> > + * @emit_sample_capture: Emit the commands in the command streamer<br> > + * for a particular gpu engine.<br> > + *<br> > + * The commands are inserted to capture the perf sample data at<br> > + * specific points during workload execution, such as before and after<br> > + * the batch buffer.<br> > + */<br> > + void (*emit_sample_capture)(struct i915_perf_stream *stream,<br> > + struct drm_i915_gem_request *request,<br> > + bool preallocate);<br> > +};<br> > +<br> <br> It seems the motivation for this following enum is mostly to deal with<br> the fact that engine->perf_srcu is set before the OA unit is configured.<br> Would it possible to set it later so that we get rid of the enum?<o:p></o:p></p> </div> </div> <p class="MsoNormal"><Sagar> I will try to make this as just binary state. This enum is defining the state of the stream. I too got confused with purpose of IN_PROGRESS.<br> SRCU is used for synchronizing stream state check.<br> IN_PROGRESS will enable us to not advertently try to access the stream vma for inserting the samples, but I guess depending on disabled/enabled should<br> suffice.<o:p></o:p></p> </blockquote> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">Hi Sagar/Lionel,<o:p></o:p></p> </div> </div> </div> </div> </blockquote> <p class="MsoNormal"><o:p> </o:p></p> </div> </div> <p class="MsoNormal">Hi Sourab,<br> <br> Thanks again for your input on this.<br> <br> <br> <o:p></o:p></p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <div> <div> <div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">The purpose of the tristate was to workaround a particular kludge of<o:p></o:p></p> </div> <div> <p class="MsoNormal">working with just enabled/disabled boolean state. I'll explain below.<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">Let's say we have only boolean state.<o:p></o:p></p> </div> <div> <p class="MsoNormal"><span style="color:#500050">i915_perf_emit_sample_capture() function would depend on</span><o:p></o:p></p> </div> <div> <p class="MsoNormal"><span style="color:#500050">stream->enabled in order to insert the MI_RPC command in RCS.</span><o:p></o:p></p> </div> <div> <p class="MsoNormal"><span style="color:#500050">If you see </span>i915_perf_enable_locked(), stream->enabled is set before<o:p></o:p></p> </div> <div> <p class="MsoNormal">stream->ops->enable(). The stream->ops->enable() function actually<o:p></o:p></p> </div> <div> <p class="MsoNormal">enables the OA hardware to capture reports, and if MI_RPC commands<o:p></o:p></p> </div> <div> <p class="MsoNormal">are submitted before OA hw is enabled, it may hang the gpu.<o:p></o:p></p> </div> </div> </div> </div> </blockquote> <p class="MsoNormal" style="margin-bottom:12.0pt"><br> Do you remember if this is documented anywhere?<br> I couldn't find anything in the MI_RPC instruction.<o:p></o:p></p> </div> </blockquote> <div> <p class="MsoNormal">Sorry, I don't happen to remember any documentation. Probably, you can check this out by submitting MI_RPC without enabling OA.<o:p></o:p></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thing is gen7_oa_enable is depending on enabled to be true.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Lionel, can I change it to :<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> if (!stream->enabled) {<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> I915_WRITE(GEN7_OACONTROL,<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> (ctx_id & GEN7_OACONTROL_CTX_MASK) |<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> (period_exponent <<<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">With stream->enabled set to true at the end of i915_perf_enable_locked.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"> <div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <div> <div> <div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">Also, we can't change the order of calling these operations inside<o:p></o:p></p> </div> <div> <p class="MsoNormal">i915_perf_enable_locked() since <span style="font-size:9.0pt;font-family:Consolas;color:#6F42C1">gen7_update_oacontrol_locked()</span><o:p></o:p></p> </div> <div> <p class="MsoNormal"><span style="font-size:9.0pt;font-family:Consolas;color:#6F42C1">function depends on stream->enabled flag to enable the OA</span><o:p></o:p></p> </div> <div> <p class="MsoNormal"><span style="font-size:9.0pt;font-family:Consolas;color:#6F42C1">hw unit (i.e. it needs the flag to be true).</span><o:p></o:p></p> </div> </div> </div> </div> </blockquote> <p class="MsoNormal"><br> We can probably work around that by passing some arguments.<br> <br> <br> <o:p></o:p></p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <div> <div> <div> <div> <p class="MsoNormal"><span style="font-size:9.0pt;font-family:Consolas;color:#6F42C1">To workaround this problem, I introduced a tristate here.</span><o:p></o:p></p> </div> <div> <p class="MsoNormal"><span style="font-size:9.0pt;font-family:Consolas;color:#6F42C1">If you can suggest some alternate solution to this problem,</span><o:p></o:p></p> </div> <div> <p class="MsoNormal"><span style="font-size:9.0pt;font-family:Consolas;color:#6F42C1">we can remove this tristate kludge here.</span><o:p></o:p></p> </div> <div> <p class="MsoNormal"><span style="font-size:9.0pt;font-family:Consolas;color:#6F42C1">Regards,</span><o:p></o:p></p> </div> <div> <p class="MsoNormal"><span style="font-size:9.0pt;font-family:Consolas;color:#6F42C1">Sourab</span><o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> </div> </div> </div> </blockquote> <p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p> </div> </blockquote> </div> <p class="MsoNormal"><o:p> </o:p></p> </div> </div> </div> </body> </html>