unit test requirements ...

[Michael Meeks]

Hi guys,

On Thu, 2013-04-04 at 12:15 +0200, Bjoern Michaelsen wrote:
> > Besides, LO is a C++ program - if you can already code in C++, why
> > would you want to switch to a different language to write a unit
> > test?
> 
> Because there are people willing to write Python tests but unable (or unwilling
> to invest the time) to write C++ tests?

	So - can we rip the language issue out of this. AFAICS no-one truly
cares about the language that the tests are written in but they do have
some pretty firm requirements that need to work for any/all of: Java,
Python, StarBasic etc.

	+ debug-ability - when the test fails - and the best tests are
	  written to fail regularly ;-) we need to be able -very-simply-
	  to get a backtrace we can work with - without a ton of
	  training, reading a dozen wiki pages, poking for obscure
	  symbols, etc.

	+ that backtrace should not have big, opaque chunks that
	  are either empty (cf. Java) or point to random / irrelevant
	  pieces of C/C++ interpreter code - Python / StarBasic &
	  others. It should allow interactive inspection of variables
	  up and down the stack.

	+ reliability and performance: new unit tests should be small,
	  fast, non-duplicative (ie. re-using existing code &
	  frameworks), readable, and must produce zero false positives.
	  Ideally they should test just one thing - though clearly our
	  bootstrapping process is a nightmare that exercises millions
	  of lines to get to the 'go'.

	+ ideally - the unit tests should run -in-the-same-process-
	  which significantly helps with the above performance,
	  debugging, reliability and more.

	Personally, flameage aside, my (by far) most preferred solution would
be not to discourage python / java / starbasic from being used for unit
tests - but instead to encourage working on making python actually
debuggable via gdb: presumably (with the new python/gdb framework ;-) 

	That is not a completely unrealistic hope / expectation ? surely ? it
would help adoption of hybrid native-code with python / StarBasic /Java
etc. across the computing universe rather than just in LibreOffice :-)

	It is still amazing to me that decades from its creation, the debugging
experience around Java from gdb is one of the foulest and least useful
things I've ever had the misfortune to use :-) - oh, wait - Mono was
just as bad, the stack traces were -still- entirely opaque and unusable
for large sections as far as I could see. Of course, in the mono case
they wrote their own custom debugger - which worked well - except for it
being a new piece of code with it's own bugs, problems, platform issues,
etc.

> But I see little reason to dogmatically write all tests in C++ as:

	It is not the test - it is the debugging of the test that is the issue.
That is something that (potentially) stops people getting their code
into LibreOffice. The higher we make that barrier - and the easier we
make it for people to raise a barrier to entry that is fundamentally
unusable / not-accessible for new people, the worse a world we live in.
"Just re-write that python stuff in C++ in order to work out why the
test fails for your new patch" is not a nice message for a new
contributor stymied by a potentially buggy & un-debug-able test.

	That is, unless we can debug them cleanly - which I hope is somewhere
we can get to without overmuch effort.

	IMHO it is not trolling to point that out; in particular I'd recommend
listening to the advice from those who have written and debugged the
most in the area of unit tests; which is neither me nor you :-)

	ATB,

		Michael.

-- 
michael.meeks at suse.com  <><, Pseudo Engineer, itinerant idiot