[Intel-gfx] [ANNOUNCE] xf86-video-intel 2.8.0

[Alan W. Irwin]

On 2009-07-31 19:12-0400 Ben Gamari wrote:

> On Fri, Jul 31, 2009 at 12:53:55PM -0700, Alan W. Irwin wrote:
>> Carl said:
>>> What I see there is lots of gtkperf microbenchmarks, which as I put
>>> forth in the blog report, don't capture realistic application behavior.
>>> So there may or may not be any real performance problem based on those
>>> numbers. It's really hard to know.
>>
>> Carl, it's a shame to see this disconnect between the Phoronix test suite
>> results and what you would like to see tested for 2D graphics.  Blogging
>> about the issue is too easily ignored. Thus, my opinion is it is long past
>> time for you or someone else from Intel (since Intel's graphics reputation
>> is on the line) to directly contact Michael Larabel to help him put together
>> some more realistic 2D tests.
>
> This is one of my issues with Phoronix. If someone wishes to put up a
> open source news site including benchmarking, that is fine. However, for
> the good of the community, they should at least take the time to
> understand the basic principles behind reproducible benchmarking.
> Several of the benchmarks I have seen on Phoronix have had fundamental
> methodological flaws. I have seen several people point out these flaws
> article comments, yet I have yet to see a single substantial change in
> testing methodology. I fail to understand why people still listen to
> Phoronix's reporting.
>
> In sum, it is not Intel's responsibility to teach Mr. Larabel about the
> limitations of microbenchmarks (even if it might do their brand some
> good).

I would agree with you if the PTS were proprietary, but it is not.  My sense
is the PTS provides a framework for any tests users or companies want to put
in there.  I further agree with you it is likely some bad tests have gotten
in, but the Intel guy's are free to improve that situation.  They are also
free to follow your advice and refuse to add decent test to the PTS which
then perpetuates the current situation which I don't think anybody likes.

>
>> I believe headlines like today's from Phoronix, "Intel Linux Graphics
>> On Ubuntu Still Flaky" are completely legitimate (and not snarky or
>> overly negative) based on Michael Larabel's results with PTS. However,
>> if there is an obvious way to change PTS (which is open source after
>> all) to give more realistic 2D testing, then that would do Intel's
>> graphics reputation a lot of much-needed good (assuming your
>> hypothesis is correct).
>>
>> Also, for another way to turn around the recent bad publicity, why doesn't
>> Intel run PTS results of their own and publish the 2D and 3D results
>> (including the exact graphics software stack you used which might well
>> differ from Ubuntu's) for each of your quarterly releases for your fastest
>> graphical chipset?
>
> Testing the fastest chipset would produce useless data --- comparing
> releases on different hardware is stamp-collecting at best and just
> plain misleading at worst. Personally, I would much rather see the Intel
> folks working on actually improving the codebase. Sure, some benchmarking
> is required to do this, but spending developer time on benchmarking just
> for publicity seems like a poor allocation of resources.

I also prefer Intel developers to spend most of their time on developing.
However, I also agree with you that benchmarking is part of that development
deal.  Where we differ is you seem to think it would take lots of developer
time to set up an automated system for both the benchmarking and publishing
the benchmark results.  Whoever is right on that, the important point is it
is essentially only a one-time cost followed by very small on-going
maintenance costs (if the automated benchmarking/publishing system were set
up right). I also view the resultant good publicity for Intel as a nice side
effect, but not the principal goal which is to identify speed
problems/improve stability with lots of standard and open tests that
interested users (e.g., the types like me that lurk on this list) can run
for themselves to compare with Intel's best test results.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state implementation
for stellar interiors (freeeos.sf.net); PLplot scientific plotting software
package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of
Linux Links project (loll.sf.net); and the Linux Brochure Project
(lbproject.sf.net).
__________________________

Linux-powered Science
__________________________