[Piglit] [PATCH] Allow one 'non-concurrent' test to run in parallel with concurrent tests again

[Dylan Baker]

On Wednesday, August 20, 2014 12:13:11 PM Michel Dänzer wrote:
> From: Michel Dänzer <michel.daenzer at amd.com>
> 
> This reverts commit acb824ddc53c446124d88e37db610a4f8c59d56c.
> 
> This decreases the runtime of the gpu.py profile from around 15 minutes to
> around 12 minutes on my machine, with no change in results.
> 
> If in the future there are tests which really can't run in parallel with any
> other tests, a new category should be added for them.

We already have such a category, non-concurrent. What has happened is a
lot of people have cut and pasted all.py entries as non-concurrent
without understanding what those flags do. I've tried several times
reverse the flag (ie, concurrent by default), but it's a staggering
amount of work.

I feel that this is just a band-aid masking the real problem, because I
suspect with the exception of glean there are probably no more than 100
tests that need to be run in serial. I'd rather see another effort like
Marek's effort a year or so ago to fix tests that should have been
marked concurrent but were not.

However, that said this is not a NAK, if others want to go this way I
won't stop it.

> 
> Signed-off-by: Michel Dänzer <michel.daenzer at amd.com>
> ---
>  framework/profile.py | 29 +++++++++++++++--------------
>  1 file changed, 15 insertions(+), 14 deletions(-)
> 
> diff --git a/framework/profile.py b/framework/profile.py
> index 5428890..1bb2b50 100644
> --- a/framework/profile.py
> +++ b/framework/profile.py
> @@ -189,8 +189,6 @@ class TestProfile(object):
>          self._pre_run_hook()
>          framework.exectest.Test.OPTS = opts
>  
> -        chunksize = 1
> -
>          self._prepare_test_list(opts)
>          log = Log(len(self.test_list), opts.verbose)
>  
> @@ -203,30 +201,33 @@ class TestProfile(object):
>              name, test = pair
>              test.execute(name, log, json_writer, self.dmesg)
>  
> -        def run_threads(pool, testlist):
> -            """ Open a pool, close it, and join it """
> -            pool.imap(test, testlist, chunksize)
> -            pool.close()
> -            pool.join()
> -
>          # Multiprocessing.dummy is a wrapper around Threading that provides a
>          # multiprocessing compatible API
>          #
>          # The default value of pool is the number of virtual processor cores
>          single = multiprocessing.dummy.Pool(1)
>          multi = multiprocessing.dummy.Pool()
> +        chunksize = 1
>  
>          if opts.concurrent == "all":
> -            run_threads(multi, self.test_list.iteritems())
> +            multi.imap(test, self.test_list.iteritems(), chunksize)
>          elif opts.concurrent == "none":
> -            run_threads(single, self.test_list.iteritems())
> +            single.imap(test, self.test_list.iteritems(), chunksize)
>          else:
>              # Filter and return only thread safe tests to the threaded pool
> -            run_threads(multi, (x for x in self.test_list.iteritems()
> -                                if x[1].run_concurrent))
> +            multi.imap(test, (x for x in self.test_list.iteritems()
> +                              if x[1].run_concurrent), chunksize)
>              # Filter and return the non thread safe tests to the single pool
> -            run_threads(single, (x for x in self.test_list.iteritems()
> -                                 if not x[1].run_concurrent))
> +            single.imap(test, (x for x in self.test_list.iteritems()
> +                               if not x[1].run_concurrent), chunksize)
> +
> +        # Close and join the pools
> +        # If we don't close and the join the pools the script will exit before
> +        # the pools finish running
> +        multi.close()
> +        single.close()
> +        multi.join()
> +        single.join()
>  
>          log.summary()
>  
> -- 
> 2.1.0
> 
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/piglit
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.freedesktop.org/archives/piglit/attachments/20140820/67e127ab/attachment-0001.sig>