[PATCH rendercheck] Report results on a per-test basis

[Martin Peres]

On 21/10/16 19:18, Eric Anholt wrote:
> Martin Peres <martin.peres at linux.intel.com> writes:
>
>> This allows a runner such as EzBench to track each test individually
>> and not limit the resolution to groups.
>>
>> This feature can be triggered by using the -r parameter.
>
> I don't really see the point of this -- you need an external runner to
> be choosing a specific test subset to run per rendercheck call, since
> the full matrix of composite tests is too long.  Once you have an
> external runner calling rendercheck per test group, all you would be
> able to do using this patch would be save a few spawns of the process,
> which doesn't seem worth it.

Just to be sure, are you suggesting I do something like this instead?

for format in "a8r8g8b8 x8r8g8b8 ... more formats"; do
	./rendercheck -f $format
	# parse the successful groups and store them, prefixed by $format so as 
I can identify regressions per format?
done

if so, then I agree that this will work for some groups of tests but not 
the ones trying the different combinations of formats, such as :

## ConjointXor linear gradient dst=a8r8g8b8, src=(1x1R a8r8g8b8, 
1.00:1.00:1.00:1.00): pass
## ConjointXor linear gradient dst=a8r8g8b8, src=(1x1R x8r8g8b8, 
1.00:1.00:1.00:1.00): pass

or the ones that do multiple sub-tests, like:

## Src Triangles a8r8g8b8: pass
## Src TriStrip a8r8g8b8: pass
## Src TriFan a8r8g8b8: pass

Indeed, I would like to know with the smallest granularity possible what 
are the results for all the sub-tests so as I can bisect as many changes 
possible instead of bisecting to the first change that broke the group 
of tests, having a double for-loop to go through all the combination of 
composite tests is orthogonal to this problem.

This was useful for me to bisect a kernel change that broke rendering 
with glamor. It would have had had less chances of succeeding without 
this patch because multiple changes might have happened.

Anyway, don't you think this is also nice from a developer perspective 
to know what test(s) actually broke in a group, instead of having to 
download rendercheck's source, patch it to add debug printfs, recompile 
and re-run? Given how piglit works and how devs love sub-tests, I think 
I am not the only one here.

Martin