[Mesa-dev] Require micro-benchmarks for performance optimization oriented patches

[Kenneth Graunke]

On Friday, November 07, 2014 12:01:20 PM Siavash Eliasi wrote:
> I know that this might sound troublesome but since there is no 
> benchmarks done by reviewers before pushing the performance optimization 
> oriented patches into master branch, I think it's as important as piglit 
> tests and necessary to ask the patch provider for simple OpenGL micro 
> benchmarks triggering the optimized code path to see if there is any 
> real benefits and make sure that it isn't degrading the performance.
> 
> Being more strict about pushing and quality assurance of these kind of 
> patches will save hours of bisecting and hair-pulling to find the root 
> cause of performance degrades.
> 
> Best regards,
> Siavash Eliasi.

For performance patches, we generally require authors to include data in their 
commit messages showing that an application benefits by some metric.  (It's 
useful for posterity too, so people can answer "what is all this code for?").

Honestly, I think I'm okay with our usual metrics like:
- Increased FPS in a game or benchmark
- Reduced number of instructions or memory accesses in a shader program
- Reduced memory consumption
- Significant cycle reduction in callgrind or better generated code
  (ideally if it's a lot of code I'd like better justification)

We've had a lot of cases of people submitting +LOC with no actual metric of 
improvement at all, and we should absolutely reject those until /some/ data is 
supplied indicating why the patch is useful.

Microbenchmarks can be helpful in demonstrating performance improvements.  But 
they can also be pretty misleading - you can make a microbenchmark run a lot 
faster, yet not impact real applications.  On the flip side, if a developer 
can show that a patch makes a real application (e.g. DOTA 2) run faster, I 
don't see any reason to require them to additionally write a microbenchmark.  
The patch is clearly justified, and developers are already overworked.

That said, if someone would like to start building an open source 
microbenchmarking suite, I could definitely see that being a valuable project.

--Ken
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.freedesktop.org/archives/mesa-dev/attachments/20141107/45a5d3af/attachment.sig>