[Mesa-dev] renderdoc-traces: like shader-db for runtime

[Eero Tamminen]

Hi,

On 20.6.2019 22.26, Eric Anholt wrote:
> Hey folks, I wanted to show you this follow-on to shader-db I've been
> working on:
> 
> https://gitlab.freedesktop.org/anholt/renderdoc-traces
> 
> For x86 development I've got a collection of ad-hoc scripts to capture
> FPS numbers from various moderately interesting open source apps so I
> could compare-perf them.  I was only looking at specific apps when they
> seemed relevant, so it would be easy to miss regressions.
> 
> Starting work on freedreno, one of the first questions I ran into was
> "does this change to the command stream make the driver faster?".  I
> don't have my old set of apps on my debian ARM systems, and even less so
> for Chrome OS.  Ultimately, users will be judging us based on web
> browser and android app performance, not whatever I've got laying around
> on my debian system.  And, I'd love to fix that "I ignore apps unless I
> think of them" thing.
> 
> So, I've used renderdoc to capture some traces from Android apps.  With
> an unlocked phone, it's pretty easy.  Tossing those in a repo (not
> shared here), I can then run driver changes past them to see what
> happens.  See
> https://gitlab.freedesktop.org/mesa/mesa/merge_requests/1134 for some
> results.
> 
> Where is this repo going from here?
> 
> - I add a runner for doing frame-to-frame consistency tests.  We could
>    catch UB in a lot of circumstances by replaying a few times and making
>    sure that results are consistent.  Comparing frames between drivers
>    might also be interesting, though for that you would need human
>    validation since pixel values and pixels lit will change on many
>    shader optimization changes.

When bisecting, ezBench[1] lists all perf and rendering changes, and
provides visual diff for *all* the rendering changes.  Without latter,
something like this would have been really hard to notice & bisect:
	https://bugs.freedesktop.org/show_bug.cgi?id=103197

Or something like this in a moving part of a long benchmark:
	https://bugs.freedesktop.org/attachment.cgi?id=134878
	https://bugs.freedesktop.org/attachment.cgi?id=134879

As to providing info on how likely driver is to be rendering something
wrong / differently instead of differences being just due to accuracy /
calculations order... Maybe something like PSNR would be useful:
	https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio
?

	- Eero

[1] https://github.com/freedesktop/ezbench


> - Need to collect more workloads for the public repo:
> 
>    - I've tried to capture webgl on Chrome and Firefox on Linux with no
>      luck. WebGL on ff is supposed to work under apitrace, maybe I could
>      do that and then replay on top of renderdoc to capture.
> 
>    - Mozilla folks tell me that firefox's WebRender display lists can be
>      captured in browser and then replayed from the WR repo under
>      apitrace or rendredoc.
> 
>    - I tried capturing Mozilla's new Pathfinder (think SVG renderer), but
>      it wouldn't play the demo under renderdoc.
> 
>    Do you have some apps that should be represented here?
> 
> - Add microbenchmarks?  Looks like it would be pretty easy to grab
>    piglit drawoverhead results, not using renderdoc.  Capturing from
>    arbitrary apps expands the scope of the repo in a way I'm not sure I'm
>    excited about (Do we do different configs in those apps?  Then we need
>    config infrastructure.  Ugh).
> 
> - I should probably add an estimate of "does this overall improve or
>    hurt perf?"  Yay doing more stats.
> 
> - I'd love to drop scipy.  I only need it for stats.t.ppf, but it
>    prevents me from running run.py directly on my targets.
> 
> 
> _______________________________________________
> mesa-dev mailing list
> mesa-dev at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/mesa-dev
>