<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
Am 24.01.23 um 09:27 schrieb Marek Olšák:<br>
<blockquote type="cite"
cite="mid:CAAxE2A4zRPr5fJdQdAoBw8=OMrm9BjJHgnzLrXoomq0o-29UBw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>A new Gallium HUD "value producer" could be added that
reads fdinfo without calling the driver.</div>
</div>
</blockquote>
<br>
That sounds good. To be honest I would have plenty of use for that.<br>
<br>
<blockquote type="cite"
cite="mid:CAAxE2A4zRPr5fJdQdAoBw8=OMrm9BjJHgnzLrXoomq0o-29UBw@mail.gmail.com">
<div dir="ltr">
<div>I still think there is merit in having this in amdgpu_drm.h
too.</div>
</div>
</blockquote>
<br>
Why?<br>
<br>
Christian.<br>
<br>
<blockquote type="cite"
cite="mid:CAAxE2A4zRPr5fJdQdAoBw8=OMrm9BjJHgnzLrXoomq0o-29UBw@mail.gmail.com">
<div dir="ltr">
<div><br>
</div>
<div>Marek<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Jan 24, 2023 at 3:13
AM Marek Olšák <<a href="mailto:maraeo@gmail.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">maraeo@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>The table of exposed driver-specific counters:<br>
</div>
<div><a
href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/gallium/drivers/radeonsi/si_query.c#L1751"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/gallium/drivers/radeonsi/si_query.c#L1751</a></div>
<div><br>
</div>
<div>Counter enums. They use the same interface as e.g.
occlusion queries, except that begin_query and end_query
save the results in the driver/CPU.<br>
</div>
<div><a
href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/gallium/drivers/radeonsi/si_query.h#L45"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/gallium/drivers/radeonsi/si_query.h#L45</a></div>
<div><br>
</div>
<div>Counters exposed by the winsys:</div>
<div><a
href="https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/gallium/include/winsys/radeon_winsys.h#L126"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/gallium/include/winsys/radeon_winsys.h#L126</a></div>
<div><br>
</div>
<div>I just need to query the counters in the winsys and
return them.<br>
</div>
<div><br>
</div>
<div>Marek<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Jan 24, 2023 at
2:58 AM Christian König <<a
href="mailto:ckoenig.leichtzumerken@gmail.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">ckoenig.leichtzumerken@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div> How are the counters which the HUD consumes
declared?<br>
<br>
See what I want to avoid is a) to nail down the
interface with the kernel on specific values and b) make
it possible to easily expose new values.<br>
<br>
In other words what we could do with fdinfo is to have
something like this:<br>
<br>
GALLIUM_FDINFO_HUD=drm-memory-vram,amd-evicted-vram,amd-mclk glxgears<br>
<br>
And the HUD just displays the values the kernel provides
without the need to re-compile mesa when we want to add
some more values nor have the values as part of the
UAPI.<br>
<br>
Christian.<br>
<br>
<div>Am 24.01.23 um 08:37 schrieb Marek Olšák:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>The Gallium HUD doesn't consume strings. It
only consumes values that are exposed as counters
from the driver. In this case, we need the driver
to expose evicted stats as counters. Each counter
can set whether the value is absolute (e.g. memory
usage) or monotonic (e.g. perf counter). Parsing
fdinfo to get the values is undesirable.<br>
</div>
<div><br>
</div>
<div>Marek<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Jan 23,
2023 at 4:31 AM Christian König <<a
href="mailto:ckoenig.leichtzumerken@gmail.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">ckoenig.leichtzumerken@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div> Let's do this as valid in fdinfo.<br>
<br>
This way we can easily extend whatever the
kernel wants to display as statistics in the
userspace HUD.<br>
<br>
Regards,<br>
Christian.<br>
<br>
<div>Am 21.01.23 um 01:45 schrieb Marek Olšák:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>We badly need a way to query evicted
memory usage. It's essential for
investigating performance problems and it
uncovered the buddy allocator disaster.
Please either suggest an alternative,
suggest changes, or review. We need it
ASAP.<br>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Marek<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue,
Jan 10, 2023 at 11:55 AM Marek Olšák <<a
href="mailto:maraeo@gmail.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">maraeo@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Tue, Jan 10, 2023 at 11:23 AM
Christian König <<a
href="mailto:ckoenig.leichtzumerken@gmail.com"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">ckoenig.leichtzumerken@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div> Am 10.01.23 um 16:28 schrieb
Marek Olšák:<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr"
class="gmail_attr">On Wed,
Jan 4, 2023 at 9:51 AM
Christian König <<a
href="mailto:ckoenig.leichtzumerken@gmail.com"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">ckoenig.leichtzumerken@gmail.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div> Am 04.01.23 um 00:08
schrieb Marek Olšák:<br>
<blockquote type="cite">
<div dir="ltr">
<div>I see about the
access now, but
did you even look
at the patch?</div>
</div>
</blockquote>
<br>
I did look at the patch,
but I haven't fully
understood yet what you
are trying to do here.<br>
</div>
</blockquote>
<div><br>
</div>
<div>First and foremost, it
returns the evicted size
of VRAM and visible VRAM,
and returns visible VRAM
usage. It should be
obvious which stat
includes the size of
another.<br>
</div>
<div><br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div> <br>
<blockquote type="cite">
<div dir="ltr">
<div> Because what
the patch does
isn't even exposed
to common drm
code, such as the
preferred domain
and visible VRAM
placement, so it
can't be in fdinfo
right now.<br>
</div>
<div><br>
</div>
<div>Or do you even
know what fdinfo
contains? Because
it contains
nothing useful. It
only has VRAM and
GTT usage, which
we already have in
the INFO ioctl, so
it has nothing
that we need. We
mainly need the
eviction
information and
visible VRAM
information now.
Everything else is
a bonus.<br>
</div>
</div>
</blockquote>
<br>
Well the main question
is what are you trying
to get from that
information? The
eviction list for
example is completely
meaningless to
userspace, that stuff is
only temporary and will
be cleared on the next
CS again.<br>
</div>
</blockquote>
<div><br>
</div>
<div>I don't know what you
mean. The returned
eviction stats look
correct and are stable
(they don't change much).
You can suggest changes if
you think some numbers are
not reported correctly.<br>
</div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div> <br>
What we could expose is
the VRAM over-commit
value, e.g. how much BOs
which where supposed to
be in VRAM are in GTT
now. I think that's what
you are looking for
here, right?<br>
</div>
</blockquote>
<div><br>
</div>
<div>The VRAM overcommit
value is "evicted_vram".<br>
</div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div> <br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Also, it's
undesirable to
open and parse a
text file if we
can just call an
ioctl.</div>
</div>
</div>
</blockquote>
<br>
Well I see the reasoning
for that, but I also see
why other drivers do a
lot of the stuff we have
as IOCTL as separate
files in sysfs, fdinfo
or debugfs.<br>
<br>
Especially repeating all
the static information
which were already
available under sysfs in
the INFO IOCTL was a
design mistake as far as
I can see. Just compare
what AMDGPU and the KFD
code is doing to what
for example i915 is
doing.<br>
<br>
Same for things like
debug information about
a process. The fdinfo
stuff can be queried
from external tools
(gdb, gputop, umr
etc...) as well which
makes that interface
more preferred.<br>
</div>
</blockquote>
<div><br>
</div>
<div>Nothing uses fdinfo in
Mesa. No driver uses sysfs
in Mesa except drm shims,
noop drivers, and Intel
for perf metrics. sysfs
itself is an unusable mess
for the PCIe query and is
missing information.</div>
<div><br>
</div>
<div>I'm not against
exposing more stuff
through sysfs and fdinfo
for tools, but I don't see
any reason why drivers
should use it (other than
for slowing down queries
and initialization).</div>
</div>
</div>
</blockquote>
<br>
That's what I'm asking: Is this
for some tool or to make some
driver decision based on it?<br>
<br>
If you just want the numbers for
over displaying then I think it
would be better to put this into
fdinfo together with the other
existing stuff there.<br>
</div>
</blockquote>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div> <br>
If you want to make allocation
decisions based on this then we
should have that as IOCTL or even
better as mmap() page between
kernel and userspace. But in this
case I would also calculation the
numbers completely different as
well.<br>
<br>
See we have at least the following
things in the kernel:<br>
1. The eviction list in the VM.<br>
Those are the BOs which are
currently evicted and tried to
moved back in on the next CS.<br>
<br>
2. The VRAM over commit value.<br>
In other words how much more
VRAM than available has the
application tried to allocate?<br>
<br>
3. The visible VRAM usage by this
application.<br>
<br>
The end goal is that the eviction
list will go away, e.g. we will
always have stable allocations
based on allocations of other
applications and not constantly
swap things in and out.<br>
<br>
When you now expose the eviction
list to userspace we will be stuck
with this interface forever.<br>
</div>
</blockquote>
<div><br>
</div>
<div>It's for the GALLIUM HUD.</div>
<div><br>
</div>
<div>The only missing thing is the
size of all evicted VRAM
allocations, and the size of all
evicted visible VRAM allocations.<br>
</div>
<div><br>
</div>
<div>1. No list is exposed. Only sums
of buffer sizes are exposed. Also,
the eviction list has no meaning
here. All lists are treated equally,
and mem_type is compared with
preferred_domains to determine where
buffers are and where they should
be.<br>
</div>
<div><br>
</div>
<div>2. I'm not interested in the
overcommit value. I'm only
interested in knowing the number of
bytes of evicted VRAM right now. It
can be as variable as the CPU load,
but in practice it shouldn't be
because PCIe doesn't have the
bandwidth to move things quickly.<br>
</div>
<div><br>
</div>
<div>3. Yes, that's true.</div>
<div><br>
</div>
<div>Marek</div>
<br>
</div>
</div>
</blockquote>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
<br>
</body>
</html>