[PATCH v4 2/5] drm/xe/eustall: Introduce API for EU stall sampling

[Harish Chegondi]

On Fri, Nov 22, 2024 at 10:18:14AM -0800, Dixit, Ashutosh wrote:
> On Wed, 20 Nov 2024 19:18:24 -0800, Dixit, Ashutosh wrote:
> >
> > On Wed, 20 Nov 2024 17:05:27 -0800, Umesh Nerlige Ramappa wrote:
> > >
> > > On Wed, Nov 20, 2024 at 11:04:28AM -0800, Dixit, Ashutosh wrote:
> > > > On Tue, 19 Nov 2024 15:59:12 -0800, Harish Chegondi wrote:
> > > >>
> > > >> > > +/**
> > > >> > > + * enum drm_xe_eu_stall_property_id - EU stall sampling input property ids.
> > > >> > > + *
> > > >> > > + * These properties are passed to the driver as a chain of
> > > >> > > + * @drm_xe_ext_set_property structures with @property set to these
> > > >> > > + * properties' enums and @value set to the corresponding values of these
> > > >> > > + * properties. @drm_xe_user_extension base.name should be set to
> > > >> > > + * @DRM_XE_EU_STALL_EXTENSION_SET_PROPERTY.
> > > >> > > + */
> > > >> > > +enum drm_xe_eu_stall_property_id {
> > > >> > > +#define DRM_XE_EU_STALL_EXTENSION_SET_PROPERTY		0
> > > >> > > +	/**
> > > >> > > +	 * @DRM_XE_EU_STALL_PROP_SAMPLE_RATE: Sampling rate
> > > >> > > +	 * in multiples of 251 cycles. Valid values are 1 to 7.
> > > >> > > +	 * If the value is 1, sampling interval is 251 cycles.
> > > >> > > +	 * If the value is 7, sampling interval is 7 x 251 cycles.
> > > >> > > +	 */
> > > >> > > +	DRM_XE_EU_STALL_PROP_SAMPLE_RATE = 1,
> > > >> >
> > > >> > What is the rate of 251 cycles? If that can be clearly defined, then at
> > > >> > first glance, I would think it's better to define this in terms of
> > > >> > frequency. The implementation can decide how to translate that to HW
> > > >> > configuration.
> > > >> >
> > > >> Since the duration of a cycle depends on the GPU clock, it can very from
> > > >> GPU to GPU. So, if there is any translation in the driver, it will have
> > > >> to be different for each GPU. I think keeping this input as a multiplier
> > > >> of cycles may be more future proof for the uAPI. I am trying to get more
> > > >> information and feedback from the user space regarding your suggestion.
> > > >> If it is feasible, I will implement in v6.
> > > >
> > > > Umesh has a point but I sort of agree with Harish because this value is
> > > > directly fed into a register. But we do need some changes:
> > > >
> > > > 1. This 251 value showing up here doesn't make any sense and needs to go.
> > > > 2. According to Bspec 64036, HW supports "127 * N" sampling rates
> > > >   (in terms of cycles), so we should support those too.
> > > > 3. Even higher sampling rates (say 10x) are being proposed for the
> > > >   future. So these should also be supported.
> > > >
> > > > So my proposal is simple, but let's see if it can be made to work. The uapi
> > > > will directly input the sampling rate in number of cycles (so the value
> > > > coming in is what the GPU freq is divided by). So e.g. if "3 * 251" is
> > > > required "3 * 251" will come in through the uapi. If UMD wants "7 * 127",
> > > > they will send in "7 * 127". The driver will internally map this value into
> > > > the "closest" sampling rate supported by HW.
> > > >
> > > > I am assuming that UMD's already know what sampling rates are supported by
> > > > a particular HW platform so they can send in the exact value they
> > > > need. Otherwise the driver can always map the value sent by userspace. Say
> > > > UMD sends a value 10, this will be mapped into "1 * 127" which is the
> > > > closest sampling rate supported to 10.
> > > >
> > > > So this way all sampling rates can be supported. UMD just says I want a
> > > > sampling rate of "GPU_freq divided by 10" and they automatically get
> > > > whatever is the closest available. They probably do need to have an idea of
> > > > what rates are supported on a particular HW platform, I am assuming they
> > > > have this information from Bspec, so they can send in exact values if they
> > > > know and driver will be able to set the exact value UMD has specified.
> 
> This is an important point here. Can we assume UMD's already know the
> sampling rates available or not? KMD can handle whatever value UMD sends in
> but UMD still has to have an idea of what sampling rates are supported by
> KMD, to be able to set an acceptable sampling rate.
> 
> Otherwise, KMD will need to expose what sampling rates are supported to
> UMD. There seem to be a couple of ways of doing this:
> 
> 1. KMD can expose the available sampling rates in drm_xe_query_eu_stall
> 
> Or,
> 
> 2. The UMD can set their sampling rate and the actual sampling rate set by
>    KMD for that stream can exposed through DRM_XE_OBSERVATION_IOCTL_INFO
>    ioctl. If UMD doesn't like the sampling rate set by KMD they can close
>    the stream and reopen a new stream. Basically they would need to iterate
>    to arrive at an acceptable sampling rate.
> 
> Or,
> 
> 3. KMD does nothing, assumes UMD's know the sampling rate "out of band" via
>    things such as Bspec.
> 
> I'm thinking maybe should expose the min and max sampling rates through 1.
> and UMD's can select something in between and KMD can map that to one of
> the sampling rates available in HW and maybe also expose the final sampling
> rate through 2.
> 
> So e.g. for now min and max exposed would be 1*251 through 7*251 (ignoring
> the 127 stuff for now). Min/max might be more future proof since HW changes
> to a different scheme, rather than have 7 or 14 fixed sampling rates they
> have now.

As of now, the driver sets a medium sampling rate as default if the user
doesn't pass any sampling rate through the properties. In this patch
series, 4 is the default sampling rate which is the mid point between 1,
the fastest sampling rate and 7, the slowest sampling rate. It would be
good to get feedback from the UMDs on which sampling rates are typically
used. As of now, Mesa is setting the sampling rate to the fastest - 1.
If the driver exposes the min and max sampling rates and sets the
mid point as the default sampling rate, the user can override with the
fastest or the slowest sampling rate or any other rate in between.

Thanks
Harish.
> 
> > >
> > > I am okay with whatever makes sense from a UMD perspective and whatever can
> > > be extended easily in future. Just the cycles as you are suggesting should
> > > be good as well.
> >
> > My suggestion is just the inverse of sampling freq, in a sense, i.e.
> >
> >	eu stall sampling freq = gpu freq / N
> >
> > So we get N through the uapi.
> >
> > > Just curious, since the gpu frequency is may vary, any idea how the UMDs
> > > map this data to their timeline?
> >
> > Note that we are dealing with the sampling freq here, the freq at which eu
> > stall data is sampled, say  1 / n * 251 or 1 / n * 127 of gpu freq. But yes
> > sampling freq can vary with gpu freq.
> >
> > But otherwise UMD's don't deal with time very much, only the IP
> > (instruction pointer). So e.g. eu stall data contains: basically IP x was
> > stalled for y cycles in this snapshot/sample. Since IP is unique, it can be
> > mapped to a line of code for an eu kernel e.g.
> >
> > So I guess there is also an approximate notion of time, so when this eu
> > kernel was sampled (or ran) the first time, IP x was stalled for y cycles,
> > when it was sampled the next time, IP w was stalled for z cycles. So you
> > would have a kind of moving window for the stall times for a particular
> > IP. So it's based on IP, rather than time directly.
> >
> > At least that's how I think it works :/