[Piglit] [PATCH v3] khr_texture_compression_astc: Enable subtest reports

[Ilia Mirkin]

On Tue, Dec 1, 2015 at 1:58 PM, Nanley Chery <nanleychery at gmail.com> wrote:
> On Tue, Dec 01, 2015 at 10:21:23AM -0500, Ilia Mirkin wrote:
>> On Tue, Dec 1, 2015 at 1:31 AM, Nanley Chery <nanleychery at gmail.com> wrote:
>> > From: Nanley Chery <nanley.g.chery at intel.com>
>> >
>> > Enable Jenkins to report the result of each individual subtest
>> > for the array and miptree ASTC tests. Modify the miptree test
>> > to only run one subset of ASTC formats at a time.
>> >
>> > v2. Modify miptree test to only check the given subtest.
>> >     Use the -subtest option to run a specific subtest.
>> > v3. Indent function arguments and misc cleanups (Ilia).
>> >     Initialize texture objects to 0 (Ilia).
>> >
>> > Signed-off-by: Nanley Chery <nanley.g.chery at intel.com>
>> > ---
>> >  tests/all.py                                       | 12 ++--
>> >  .../khr_compressed_astc-miptree.c                  | 84 +++++++++-------------
>> >  2 files changed, 42 insertions(+), 54 deletions(-)
>> >
>> > diff --git a/tests/all.py b/tests/all.py
>> > index cfafa71..29ee6a9 100644
>> > --- a/tests/all.py
>> > +++ b/tests/all.py
>> > @@ -4223,12 +4223,16 @@ with profile.group_manager(
>> >           PiglitGLTest,
>> >           grouptools.join('spec', 'khr_texture_compression_astc')) as g:
>> >      g(['arb_texture_compression-invalid-formats', 'astc'], 'invalid formats')
>> > -    g(['khr_compressed_astc-array_gl'], 'array-gl')
>> > -    g(['khr_compressed_astc-array_gles3'], 'array-gles')
>> >      g(['khr_compressed_astc-basic_gl'], 'basic-gl')
>> >      g(['khr_compressed_astc-basic_gles2'], 'basic-gles')
>> > -    g(['khr_compressed_astc-miptree_gl'], 'miptree-gl')
>> > -    g(['khr_compressed_astc-miptree_gles2'], 'miptree-gles')
>> > +
>> > +    for subtest in ('odd', 'even'):
>> > +        g(['khr_compressed_astc-array_gl', '-subtest', subtest])
>> > +        g(['khr_compressed_astc-array_gles3', '-subtest', subtest])
>> > +
>> > +    for subtest in ('ldr', 'srgb', 'hdr'):
>> > +        g(['khr_compressed_astc-miptree_gl', '-subtest', subtest])
>> > +        g(['khr_compressed_astc-miptree_gles2', '-subtest', subtest])
>>
>> I still don't understand why you need this... what's wrong with the
>> current way? All the subtests should get run, and be properly
>> reported, without needing to be included one-by-one here. Does that
>> not happen?
>
> This enables the Jenkins CI system to report which specific subtest is
> failing - without having to inspect the error message to determine the

Wow, that seems like a pretty big failing for the jenkins integration
-- So basically if you use jenkins, subtests are useless? Should we be
discouraging them? I don't suppose fixing jenkins is an option?

> cause after every run. One known failure is the sRGB miptree subtest.
> It's unresolved whether or not the issue lies within HW or the codec-
> provided texture. I would like to be notified when the other modes are
> failing as that would indicate something has broken in mesa.

Hm, well my experience with Qualcomm Adreno A420 is that (a) I get
black instead of the error color when loading HDR textures, and (b) I
need much wider color tolerances -- IIRC I said color had 6 bits of
resolution. Once I did that, the ldr and srgb tests both passed. Both
are attributable to incorrect programming of the hardware though.
There's no easy way (currently) to run piglit tests against android
userspace, and even if there were, their drivers tend to be pretty
weak in terms of ... working.

I couldn't get the array tests to work since I think they implicitly
depended on some higher GL *and* GLES version than what the freedreno
driver provides [GL 3.1, GLES 3.0].

  -ilia