unit test requirements ...

[Bjoern Michaelsen]

Hi,

On Thu, Apr 04, 2013 at 11:42:46AM +0100, Michael Meeks wrote:
> > But I see little reason to dogmatically write all tests in C++ as:
> 
> 	It is not the test - it is the debugging of the test that is the issue.
> That is something that (potentially) stops people getting their code
> into LibreOffice. The higher we make that barrier - and the easier we
> make it for people to raise a barrier to entry that is fundamentally
> unusable / not-accessible for new people, the worse a world we live in.

Sooo, IMHO there are multiple ways to use tests and that does different things.
_If_ a test or test suite is run on every change submitted to e.g. gerrit and
having an influence on the 'verified' result of the tinderbox, than it has to
be stable and passing on master too. And if _that_ is the case, assuming the
patch is not some 10k LOC diff, it shouldnt be had to guess what the root of
the trouble is -- even without debugging. Limiting the scope is king here: If
the change is small and the test is readable, you will rarely need debugging.
If you need it, it shows other troubles (commit too big, test broken or wrong).

> "Just re-write that python stuff in C++ in order to work out why the
> test fails for your new patch" is not a nice message for a new
> contributor stymied by a potentially buggy & un-debug-able test.

Hey, "that test is stupid, removed" is a valid resolution of a test failure, if
that is the case.

_If_ tests are not run on every commit (and once we have to luxury of a huge test
set we might need to consider that) and we do something like running them every
week (like e.g. bug docs stuff), only _then_ you lose the locality/limited scope
and then you need debugability. But in that case:
- its not blocking contributions
- its worth replicating a failed test in C++ then
- you lose nothing, you win something => Good Thing(tm) even then

> 	That is, unless we can debug them cleanly - which I hope is somewhere
> we can get to without overmuch effort.

Debugability is mostly a red herring if you have good locality (just like a
good bibisect makes fixing regressions a lot easier).

 
> 	IMHO it is not trolling to point that out;

Do we really need this strawman? Nobody suggested that.

Best,

Bjoern