[Intel-gfx] [ANNOUNCE] xf86-video-intel 2.8.0

[Ben Gamari]

On Fri, Jul 31, 2009 at 12:53:55PM -0700, Alan W. Irwin wrote:
> Carl said:
> >What I see there is lots of gtkperf microbenchmarks, which as I put
> >forth in the blog report, don't capture realistic application behavior.
> >So there may or may not be any real performance problem based on those
> >numbers. It's really hard to know.
> 
> Carl, it's a shame to see this disconnect between the Phoronix test suite
> results and what you would like to see tested for 2D graphics.  Blogging
> about the issue is too easily ignored. Thus, my opinion is it is long past
> time for you or someone else from Intel (since Intel's graphics reputation
> is on the line) to directly contact Michael Larabel to help him put together
> some more realistic 2D tests.  

This is one of my issues with Phoronix. If someone wishes to put up a
open source news site including benchmarking, that is fine. However, for
the good of the community, they should at least take the time to
understand the basic principles behind reproducible benchmarking.
Several of the benchmarks I have seen on Phoronix have had fundamental
methodological flaws. I have seen several people point out these flaws
article comments, yet I have yet to see a single substantial change in
testing methodology. I fail to understand why people still listen to
Phoronix's reporting.

In sum, it is not Intel's responsibility to teach Mr. Larabel about the
limitations of microbenchmarks (even if it might do their brand some
good).   

> I believe headlines like today's from Phoronix, "Intel Linux Graphics
> On Ubuntu Still Flaky" are completely legitimate (and not snarky or
> overly negative) based on Michael Larabel's results with PTS. However,
> if there is an obvious way to change PTS (which is open source after
> all) to give more realistic 2D testing, then that would do Intel's
> graphics reputation a lot of much-needed good (assuming your
> hypothesis is correct).
> 
> Also, for another way to turn around the recent bad publicity, why doesn't
> Intel run PTS results of their own and publish the 2D and 3D results
> (including the exact graphics software stack you used which might well
> differ from Ubuntu's) for each of your quarterly releases for your fastest
> graphical chipset?  

Testing the fastest chipset would produce useless data --- comparing
releases on different hardware is stamp-collecting at best and just
plain misleading at worst. Personally, I would much rather see the Intel
folks working on actually improving the codebase. Sure, some benchmarking
is required to do this, but spending developer time on benchmarking just
for publicity seems like a poor allocation of resources.

> That would give a good test of the stability of your
> recommended graphics stack and also give a good indication (assuming you
> make the appropriate 2D PTS changes) of how you are doing with regard to the
> speed issues.  It would also let users try out exactly the same PTS test
> themselves for their particular Intel chipset and graphical software stack
> to see how they are doing (stability and speed) relative to your fastest
> graphical chipset with preferred graphics software stack.
> 
> Alan