[PATCH weston] tests: make signal other than ABRT a hard failure

[Pekka Paalanen]

On Thu, 24 Jan 2013 16:24:14 -0500
Kristian Høgsberg <hoegsberg at gmail.com> wrote:

> On Tue, Dec 18, 2012 at 05:30:16PM +0200, Pekka Paalanen wrote:
> > We handle FAIL_TEST tests by simply inverting the success flag. The
> > problem with this is, that if a FAIL_TEST fails by a SIGSEGV, it will be
> > interpreted as passed. However, no code should ever cause a SEGV, or any
> > other signal than ABRT. And even ABRT only in the case of an assert()
> > that is meant to fail. We would probably need more sophistication for the
> > FAIL_TEST cases.
> >
> > For now, just interpret any other signal than ABRT as a hard failure,
> > regardless whether it is a TEST or FAIL_TEST. At least segfaults do not
> > cause false passes anymore.
> 
> I could imagine that we'd want to test for various expected failure
> modes, but we'd probably want to do something more sophisticated for
> that.  Like FAIL_TEST(name, FAIL_TEST_SIGSEGV) or something like that.

I can't really imagine any test where we would expect to get a SEGV and
run the default signal handler, or any other fatal signal for that
matter. Even ABRT should probably not be treated like we do, it's just
a shorthand for misusing assert(). I say misusing, because we assert()
on unexpected failures, and the misuse is to assert() on expected
failures, which means we cannot distinguish an unexpected and expected
failure. Nothing should really be triggering an assert() on purpose in
any tested components, since that is not really testing the component
but the assert itself. Abnormal process termination can hardly be
expected behaviour of a software component. If we expect a certain
signal at a certain point, we can install a signal handler for it, and
write helpers to make it easy.

I know I added the FAIL_TEST, and I am a little regretting it now. It
probably should only be used by tests that test the test harness's
automatic checks only, like fd or memory leaks.

One more thing to the TODO...


Thanks,
pq