parallelizing crashtest runs (was: minutes of ESC call ...)

[Michael Meeks]

Hi Christian,

On Fri, 2014-10-31 at 14:23 +0100, Christian Lohmaier wrote:
> When I played with the crashtest setup I noticed some limitations in
> the current layout of the crashtest-setup that prevents just using
> lots of cores/high parallelism to get faster results.

	Oh - these sound a bit silly =)

> The problem is that it is parallelized per directory, but the amount
> of files in a directory is not evenly distributed at all. So when the
> script decides to start odt tests last, the whole set of odt files
> will only be tested in one thread, leaving the other CPU-cores idling
> around with nothing to do.

	Interesting; if we know how many cores we have, surely we can just get
each thread to do a 'readdir' and divide that into <n> chunks - and
tackle the N'th of those (?) Or is the reason we do that to make
stitching together the reports simpler ?

> Didn't look into the overall setup to know whether just segmenting the
> large directories into smaller ones is easy to do or not (i.e instead
> of having one odt dir with 10500+ files, have 20 with ~ 500 each.

	Presumably there is no real reason to do anything odd to the
file-system - we can partition the work in whatever way seems best (?)
with some better, but still simple algorithm for partitioning /
reporting ? but - honestly, it's no use asking me - this is Markus' baby
- I'm sure he has a plan =)

	ATB,

		Michael.

-- 
 michael.meeks at collabora.com  <><, Pseudo Engineer, itinerant idiot