[Piglit] Rework profiles

[Ilia Mirkin]

Picking a random group:

# Group ARB_shader_objects
with profile.group_manager(
        PiglitGLTest,
        grouptools.join('spec', 'ARB_shader_objects')) as g:
    g(['arb_shader_objects-getuniform'], 'getuniform', run_concurrent=True)
    g(['arb_shader_objects-bindattriblocation-scratch-name'],
      'bindattriblocation-scratch-name', run_concurrent=True)
    g(['arb_shader_objects-getactiveuniform-beginend'],
      'getactiveuniform-beginend', run_concurrent=True)
    g(['arb_shader_objects-getuniformlocation-array-of-struct-of-array'],
      'getuniformlocation-array-of-struct-of-array', run_concurrent=True)
    g(['arb_shader_objects-clear-with-deleted'],
      'clear-with-deleted', run_concurrent=True)
    g(['arb_shader_objects-delete-repeat'], 'delete-repeat',
      run_concurrent=True)

So, to start, I really like this approach. (IIRC I even mentioned at
one point that I wanted to do something like this, but time has
been... lacking.)

However there's further redunancies. e.g. every test starts with
arb_shader_objects-, and takes no args. Secondly, every test in this
group is concurrent. Would be nice to encode that into the group
manager, IMO, so that each line would look like

g('getuniform')

And everything else would be implied. Thoughts?

[Obviously you need to retain the configurability to do crazy things
for the few groups that require it.]

On Tue, Jan 27, 2015 at 2:58 PM, Dylan Baker <baker.dylan.c at gmail.com> wrote:
> This is a very large, very invasive rework of the way that test profiles
> work, hopefully for the better. The goal was to create a simple syntax
> that discouraged variation, and limited accidents. What I ended up with
> is a two fold change in the way tests are added.
>
> First, shader tests, glslparser tests, and asmparser tests are
> automatically added to the profile with *zero* modification to all.py (A
> similar setup could be used by cl.py, but I wanted the input of the cl
> devs, since it would mean either renaming or moving tests). These tests
> use their directory structure to determine what groups they belong to.
>
> This necessitated another change, treating all group and test names as
> case insensitive. This will also fix the problem on case sensitive file
> systems of having tests split across groups in the html summary.
>
> The second change it the removal of add_plain_test and
> add_concurrent_test, they are replaced by a method in TestProfile that
> gives an 'adder' function, which encompasses all the power of
> add_concurrent_test and add_plain_test, but also has all of the power of
> adding tests directly to the profile.
>
> The third change is the removal of TestProfile.tests, the nested
> dictionary structure which was flattened before running, and replacing
> it with the use of TestProfile.test_list, a dictionary of string:Test
> values that assumes that the tests are already formatted appropriately,
> this series of changes makes extensive use of the grouptools module to
> ensure future flexibility in this regard.
>
> Patches 1-2 are out as a separate series.
> Patches 3-5 are groundwork changes
> Patch 6 adds two new profiles that I have carried around for some time
> and are very useful for development
> Patches 7-30 Are the bulk of the series.
>
> Patches 16-18 will need to be merged into one mega-patch to allow clean
> bisect. I have tried to come up with a better way to do this, but can't
> come up with any other solution that allows bisecting all portions of
> piglit except the mega patch
>
> The end result, even with additional testing and two new results
> versions (which add a lot of code for updating and validating), is ~+500
> LOC, even with my efforts to wrap lines better in all.py then they were
> previously.
>
> This is available at my github:
> https://github.com/dcbaker/piglit rfc/flat-test-list
>
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/piglit