[Intel-gfx] Pixel-perfect frame checks in IGT Chamelium tests and CRC

[Paul Kocialkowski]

On Thu, 2017-06-15 at 13:37 -0400, Lyude Paul wrote:
> JFYI: Hardcoded CRCs are fine I'm pretty sure, but I might be wrong. As
> well, put the chamelium in a metal box. The way the IO board is hooked
> up is not really the way something running DP should be hooked up, so
> it's very susceptible to electromagnetic interference. This will
> usually cause CRC mismatches with DP.

I don't think that is the case here, as pixel-to-pixel comparison succeeds
reliably. I also get the same hashes (that don't match the hardcoded ones)
everytime I run the test. I think the reference picture I have is just different
from the one you hardcoded the CRCs for, as I pointed out (font rendering cannot
be trusted, among other things that can change).

This is why I think we should calculate the CRC instead of using hardcoded ones.

> This being said however, I think we should have some better functions
> for the chamelium for doing frame comparisons, mainly something that
> does fuzzy frame comparisons for stuff like VGA.

Agreed. I will most likely look into that next week.

> On Thu, 2017-06-15 at 16:57 +0300, Paul Kocialkowski wrote:
> > Hi,
> > 
> > So far, there are two ways of testing for pixel-perfect frames using
> > the
> > Chamelium that are in IGT. The first one grabs a full frame from the
> > Chamelium
> > and compares it pixel-to-pixel with the cairo reference, which works
> > well for
> > DP/HDMI.
> > 
> > For VGA, this is probably not the case (because the link is
> > analogue). In that
> > case, I will look into implementing some fuzzy testing, probably
> > inspired by
> > what piglit (probably) does to compare output frames with references.
> > 
> > For pixel-perfect testing, grabbing a full frame and testing it with
> > memcmp
> > comes with a significant time penalty (about 2 seconds for 1080p).
> > The Chamelium
> > also provides a CRC mechanism that is faster and does not require
> > retrieving the
> > frame, that IGT currently also supports. It compares the CRC
> > calculated by the
> > Chamelium (implemented in the HDL) with a hardcoded reference value.
> > 
> > This approach currently fails for me (the values I get don't match
> > the hardcoded
> > reference). There are reasons why it is not really reasonable: fonts
> > rendering
> > may change between machines (e.g. use of anti-aliasing) and cairo
> > version
> > changes could introduce slight rendering changes too (not to mention
> > changes in
> > the test pattern itself). So instead of comparing the CRC with a
> > hardcoded
> > reference value, I think it would make a lot more sense to actually
> > calculate
> > the CRC based on the cairo image that is the actual reference (and
> > that we
> > should assume may change between runs/machines).
> > 
> > I am currently looking into the CRC calculation mechanism used by the
> > Chamelium
> > and trying to reproduce it in C code. Is this a known algorithm for
> > which a
> > reference/optimized implementation exists, or something custom that
> > the folks
> > over at Google came up with?
> > 
> > Any thoughts, comments or suggestions?
> > 
> > Cheers!
> > 
-- 
Paul Kocialkowski <paul.kocialkowski at linux.intel.com>
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo