[Mesa-dev] renderdoc-traces: like shader-db for runtime

[Rob Clark]

On Thu, Jun 20, 2019 at 12:26 PM Eric Anholt <eric at anholt.net> wrote:
>
> Hey folks, I wanted to show you this follow-on to shader-db I've been
> working on:
>
> https://gitlab.freedesktop.org/anholt/renderdoc-traces
>
> For x86 development I've got a collection of ad-hoc scripts to capture
> FPS numbers from various moderately interesting open source apps so I
> could compare-perf them.  I was only looking at specific apps when they
> seemed relevant, so it would be easy to miss regressions.
>
> Starting work on freedreno, one of the first questions I ran into was
> "does this change to the command stream make the driver faster?".  I
> don't have my old set of apps on my debian ARM systems, and even less so
> for Chrome OS.  Ultimately, users will be judging us based on web
> browser and android app performance, not whatever I've got laying around
> on my debian system.  And, I'd love to fix that "I ignore apps unless I
> think of them" thing.
>
> So, I've used renderdoc to capture some traces from Android apps.  With
> an unlocked phone, it's pretty easy.  Tossing those in a repo (not
> shared here), I can then run driver changes past them to see what
> happens.  See
> https://gitlab.freedesktop.org/mesa/mesa/merge_requests/1134 for some
> results.
>
> Where is this repo going from here?
>
> - I add a runner for doing frame-to-frame consistency tests.  We could
>   catch UB in a lot of circumstances by replaying a few times and making
>   sure that results are consistent.  Comparing frames between drivers
>   might also be interesting, though for that you would need human
>   validation since pixel values and pixels lit will change on many
>   shader optimization changes.
>
> - Need to collect more workloads for the public repo:
>
>   - I've tried to capture webgl on Chrome and Firefox on Linux with no
>     luck. WebGL on ff is supposed to work under apitrace, maybe I could
>     do that and then replay on top of renderdoc to capture.

perhaps worth a try capturing these on android?

I have managed to apitrace chromium-browser in the past.. it ends up a
bit weird because there are multiple contexts, but apitrace has
managed to replay them.  Maybe the multiple ctx thing is confusing
renderdoc?

(tbh I've not really played w/ renderdoc yet.. I should probably do so..)

>   - Mozilla folks tell me that firefox's WebRender display lists can be
>     captured in browser and then replayed from the WR repo under
>     apitrace or rendredoc.
>
>   - I tried capturing Mozilla's new Pathfinder (think SVG renderer), but
>     it wouldn't play the demo under renderdoc.
>
>   Do you have some apps that should be represented here?
>
> - Add microbenchmarks?  Looks like it would be pretty easy to grab
>   piglit drawoverhead results, not using renderdoc.  Capturing from
>   arbitrary apps expands the scope of the repo in a way I'm not sure I'm
>   excited about (Do we do different configs in those apps?  Then we need
>   config infrastructure.  Ugh).
>
> - I should probably add an estimate of "does this overall improve or
>   hurt perf?"  Yay doing more stats.
>
> - I'd love to drop scipy.  I only need it for stats.t.ppf, but it
>   prevents me from running run.py directly on my targets.

thoughts about adding amd_perfcntr/etc support?  I guess some of the
perfcntrs we have perhaps want some post-processing to turn into
usuable numbers, and plenty of them we don't know much about what they
are other than the name.  But some of them are easy enough to
understand (like # of fs ALU cycles, etc), and being able to compare
that before/after shader optimizations seems useful.

Also, it would be nice to have a way to extract "slow frames" somehow
(maybe out of scope for this tool, but related?).. ie. when framerate
suddenly drops, those are the frames we probably want to look at more
closely..

BR,
-R