[Intel-gfx] [PATCH i-g-t v3 4/4] chamelium: Dump obtained and reference frames to png on crc error

[Martin Peres]

On 10/07/17 17:11, Paul Kocialkowski wrote:
> On Mon, 2017-07-10 at 16:56 +0300, Martin Peres wrote:
>> On 10/07/17 15:06, Paul Kocialkowski wrote:
>>> On Mon, 2017-07-10 at 13:33 +0300, Martin Peres wrote:
>>>> On 10/07/17 13:31, Paul Kocialkowski wrote:
>>>>> On Thu, 2017-07-06 at 17:57 -0400, Lyude Paul wrote:
>>>>>>
>>>>>> As well, one advantage we do have here from the chamelium end
>>>>>> is
>>>>>> that
>>>>>> you can only really be screen grabbing from one port at a
>>>>>> time. So
>>>>>> you
>>>>>> could actually just track stuff internally in the
>>>>>> igt_chamelium
>>>>>> API
>>>>>> and when a user tries to download a framedump that we've
>>>>>> already
>>>>>> downloaded, we can just hand them back a cached copy of it.
>>>>>
>>>>> I forgot to answer this point. I think this bring way too much
>>>>> overhead
>>>>> and is not really necessary anyway. With the solution I proposed
>>>>> in
>>>>> my
>>>>> previous email on this thread, the two wrapper functions (one
>>>>> for
>>>>> CRC,
>>>>> one for analogue frame comparison) will either dump the frame
>>>>> for
>>>>> CRC
>>>>> comparison (because it was never dumped before) or use the
>>>>> provided
>>>>> one
>>>>> for analogue comparison, so there is no particular need to track
>>>>> what
>>>>> was or wasn't dumped before.
>>>>
>>>> No need to track, just encode it in the filename:
>>>> $test-$subtest-error-crc-$crc.png
>>>>
>>>> Just do not override existing files, and you are done :)
>>>
>>> I suppose it would be best to have predictible filenames (so not
>>> including the crc) to make it easier to grab the frame for an
>>> automated
>>> testing framework, right? >
>>> So what about: frame-$test-$subtest-$qualifier.png, with $qualifier
>>> being either "reference" or "dump". I don't think it's necessary to
>>> indicate whether the error comes from crc or analogue frame testing:
>>> this will already be contained in the subtest name.
>>>
>>
>> Well, predictable is actually problematic, since automated systems
>> try
>> to reproduce issues, and your idea will lead to overriding (which is
>> only not a problem if the CRC is the same).
> 
> That is, unless the automated system moves the file around from the
> predictable name to whatever suits it best (that may include which
> run the result is part of).

In this case, I would like to see a way to add a prefix, so as we can 
avoid having another step to move files afterwards.

> 
>> In the end, what we want is to say in the logs what are the files
>> (reference and dump). We'll need to agree on a format, so as an
>> automated system can pick it up :)
> 
> Parsing logs sound like a hackish way to do things quite honestly. Just
> having a predictable name and moving the file wherever relevant seems a
> lot easier on all sides.
> 
> Also, since we're making the code for frame dumping independent from
> whether it comes from crc or full frame testing, it doesn't play out too
> well to carry the crc result until the point of writing the png file.
> 

I don't like the idea of not de-duplicating images. Let's say that we 
have 20 frames that fail, it takes 1 month to fix the issue and we have 
10 runs per day, that would amount to 9.3G of redundant data (for a 
frame being 1.5MB), instead of 30 MB.

So, maybe we could have a compromise. Images are stored in $CRC.png and 
new files $prefix-frame-$test-$subtest-$qualifier are generated, 
containing the path to the right file. This is how we solved this issue 
for image comparison in ezbench. This way, we only add 4K per image 
instead of 1.5 MB.