[PATCH v3] drm/xe/pmu: Add GT frequency events
Belgaumkar, Vinay
vinay.belgaumkar at intel.com
Mon Mar 24 22:14:55 UTC 2025
-----Original Message-----
From: Vivi, Rodrigo <rodrigo.vivi at intel.com>
Sent: Monday, March 24, 2025 1:10 PM
To: De Marchi, Lucas <lucas.demarchi at intel.com>
Cc: Dixit, Ashutosh <ashutosh.dixit at intel.com>; Belgaumkar, Vinay <vinay.belgaumkar at intel.com>; intel-xe at lists.freedesktop.org; Tauro, Riana <riana.tauro at intel.com>
Subject: Re: [PATCH v3] drm/xe/pmu: Add GT frequency events
On Mon, Mar 24, 2025 at 02:07:32PM -0500, Lucas De Marchi wrote:
> On Mon, Mar 24, 2025 at 09:24:47AM -0700, Ashutosh Dixit wrote:
> > On Sat, 22 Mar 2025 19:47:04 -0700, Lucas De Marchi wrote:
> > >
> > > On Sat, Mar 22, 2025 at 02:06:09PM -0700, Ashutosh Dixit wrote:
> > > > On Sat, 22 Mar 2025 05:57:12 -0700, Lucas De Marchi wrote:
> > > >>
> > > >> On Fri, Mar 21, 2025 at 03:45:21PM -0700, Belgaumkar, Vinay wrote:
> > > >> >
> > > >> > On 3/13/2025 2:45 PM, Lucas De Marchi wrote:
> > > >> >> On Tue, Mar 11, 2025 at 05:14:08PM -0700, Vinay Belgaumkar wrote:
> > > >> >>> Define PMU events for GT frequency (actual and requested).
> > > >> >>> This is a port from the i915 driver implementation, where
> > > >> >>> an internal timer is used to aggregate GT frequencies over certain fixed interval.
> > > >> >>> Following PMU events are being added-
> > > >> >> ^
> > > >> >> why do you use "-" instead of ":"?
> > > >> >>
> > > >> >>>
> > > >> >>> xe_0000_00_02.0/gt-actual-frequency/ [Kernel
> > > >> >>> PMU event]
> > > >> >>> xe_0000_00_02.0/gt-requested-frequency/ [Kernel
> > > >> >>> PMU event]
> > > >> >>>
> > > >> >>> Standard perf commands can be used to monitor GT frequency-
> > > >> >>> $ perf stat -e
> > > >> >>> xe_0000_00_02.0/gt-requested-frequency,gt=0/ -I1000
> > > >> >>>
> > > >> >>> 1.001175175 700 M
> > > >> >>> xe/gt-requested-frequency,gt=0/
> > > >> >>> 2.005891881 703 M
> > > >> >>> xe/gt-requested-frequency,gt=0/
> > > >> >>> 3.007318169 700 M
> > > >> >>> xe/gt-requested-frequency,gt=0/
> > > >> >>>
> > > >> >>> Actual frequencies will be reported as 0 when GT is in C6.
> > > >> >>
> > > >> >>
> > > >> >> I think we need to document somewhere, but at the very least
> > > >> >> in the commit message what the event count actually is. Let
> > > >> >> me see if I get this right: if userspace is sampling every
> > > >> >> 1sec, and assuming the gpu is at 700MHz for the first 0.5sec
> > > >> >> and at 1.4 GHz, userspace should expect to see ~1050 as the
> > > >> >> value. Correct? I find this frequency handling very
> > > >> >> different from anything else reported via perf. Other than i915, are there any other cases you know of?
> > > >> >
> > > >> > Yes, user will see a gradual ramp. One thing that could work
> > > >> > is something along these lines:
> > > >> >
> > > >> > @@ -324,7 +327,10 @@ static void xe_pmu_event_update(struct
> > > >> > perf_event
> > > >> > *event)
> > > >> > new = __xe_pmu_event_read(event);
> > > >> > } while (!local64_try_cmpxchg(&hwc->prev_count,
> > > >> > &prev, new));
> > > >> >
> > > >> > - local64_add(new - prev, &event->count);
> > > >> > + if (is_gt_frequency_event(id))
> > > >> > + local64_add(new, &event->count);
> > > >> > + else
> > > >> > + local64_add(new - prev, &event->count);
> > > >> >
> > > >> > This will give us instantaneous values and will not need the
> > > >> > use of an internal timer. Should be ok to do it this way?
> > > >>
> > > >> yes, I think it'd be preferred. It's much simpler and don't
> > > >> prevent us from eventually adding an avg_* event if the needs
> > > >> arise. Which would also be clearer on what that is.
> > > >
> > > > Instantaneous values would show 0 if the sampling instant landed
> > > > when gt is
> > >
> > > instantaneous values make much more sense from the perf point of
> > > view IMO. perf is **sampling** them - other than the same thing in
> > > i915, I've never seen an event that returns an average (not even
> > > documenting it as such).
> > >
> > > What would also make sense would be to support perf-record so the
> > > samples are recorded in the buffer that is passed to userspace. In
> > > this case setting up a timer to save them (since we don't have
> > > interrupt for our pmu) would be nice.
> >
> > Afaik, typically userspace calls into the kernel for PMU counters
> > relatively infrequently, say once a second (1 Hz). However, if this
> > low rate does not yield good freq values (say freq is showing lots
> > of 0's), now
>
> why do you say it's not getting good value? is 0 not a good value?
> Should the kernel be returning something else instead of doing an
> average with "lots of zeros" and returning it?
well, zero is not a good value because it is not real. The register is zero because the GT is sleeping, but that is not the accurate thing.
but tbh we are already taking this decision to our sysfs interface:
$ cat act_freq
0
But well, for this instantaneous it may be good... we just need to educate the consumers to see the zero as gpu sleeping in a chart.
but zero is an outlier to the average of the frequencies that it will bring the mean to a very lower levels and the charts collected with PMU will be so out of shape to the point it might be useless...
True. Better to stick with instantaneous value in this case.
>
> > userspace will need to call into the kernel more frequently, say 10
> > Hz to
> > 100 Hz, just to get good freq values (all other counters are fine at
> > 1
>
> It should be perfectly fine for an application to setup a 10Hz timer
> and getting it multiple times if it wants to average.... I don't think
> it needs to be at the hundreds frequency.
>
> > Hz). This increased rate of calling into the kernel may be ok for
> > 'perf record' but not sure what happens to say 'gputop'.
>
> oh no, perf-record works diferently... it's not calling into the
> kernel with that frequency. It's perf-stat that works that way.
>
> For perf-record it more like kernel pushing samples in a ringbuffer,
> shared with userspace, and then waking it. We (currently) don't
> support that mode in our implementation:
>
> static int xe_pmu_event_init(struct perf_event *event) {
> ...
> /* unsupported modes and filters */
> if (event->attr.sample_period) /* no sampling */
> return -EINVAL;
> ...
> }
>
>
> >
> > I think considerations like this went into i915 deciding to average
> > in the kernel.
>
> as you noticed, maybe there was actually a lack of thought on the
> matter and then later it couldn't be changed anymore... ? I saw
> several odd things in the i915 side. Exposing something with a Hz unit
> is one of them.
>
> >
> > Also, if we are going to expose instantaneous freq, if gt is in C6
> > (when userspace calls into the kernel to sample), could we save and
> > return the freq /before/ gt went into C6 (to avoid returning 0 freq,
> > and returning the last freq as it was when workload was running)?
> > This might be sufficient and remove the need to sample more
> > frequently, though not sure how complex to implement.
>
> if that data would make more sense for userspace, then I think it
> should be ok. So, let's say gt in in C6, what does make more sense for
> gputop-like and turbostat?
>
> 1) read 0 from the kernel and report 0 to the user
> 2) read 0 from the kernel and report min_freq to the user (min_freq they
> can read once from from sysfs)
> 3) read min_freq from the kernel
> 4) read the last frequency before going into C6
The GT requested frequency will already show us the last requested value. Assuming no throttling, there is a good chance this is what the GT was at. Besides, KMD does not control when GT goes to C6, GuC does the idle indication (since we are using GuC RC).
Thanks,
Vinay.
>
> Lucas De Marchi
>
> >
> > >
> > > > in C6. Also instantaneous values are already available via sysfs.
> > >
> > > with the caveat of a slower interface, but yeah... that's why I
> > > was asking earlier: do we really need this if it's already
> > > available via sysfs? I was told turbostat wants to consolidate all
> > > the data to be read via perf instead of reading a bunch of files from sysfs.
> >
> > OK.
> >
> > >
> > > > Also, the way i915 calculates the average freq is itself
> > > > controversial. i915 computes average freq when gt is awake and
> > > > disregards those intervals when gt is in C6. There was this
> > > > patch to suggest that average freq also take into account those intervals when gt is in C6:
> > > >
> > > > https://patchwork.freedesktop.org/series/109372/
> > > >
> > > > But it was not accepted since it would change uapi.
> > >
> > > one more reason not to fall in the same trap. Let userspace decide
> > > if it wants the average or not. If it'd need to sample too
> > > frequently, then I think we need to look at supporting perf-record
> > > rather than doing the average (which is actually an integration,
> > > using the rectangle method, and then dividing by the perf_read
> > > delta time). At the very least, rename the event to make the avg explicit.
> >
> > OK, we can return instantaneous freq for now and if that doesn't
> > work out, add an average freq later?
> >
> > >
> > > Lucas De Marchi
> >
> > Thanks.
> > --
> > Ashutosh
More information about the Intel-xe
mailing list