[Telepathy] Improving our testing by generating the scenarios

[Sjoerd Simons]

Hi,

  So for a while i've been pondering how we can improve the way we write tests.
  Mostly for Wockey (the new of Gibber), because i want that to have very very
  comprehensive tests. And a big part of making that happen is making it easy
  to write tests :)

  To start of with a real story, this weekend i was hacking on Empathy's voip
  support a bit more and specifically ensuring that the upcoming Gabble release
  works well with it, when i noticed that Gabble gets the creator attribute
  wrong when creating a new content _if_ it wasn't the creator of the session.
  Now easy enough to add tests for that you'd think, just add a check for the
  attribute when gabble sends a content-add. Unfortunately this means digging
  through all the tests looking for cases where gabble does the content-add and
  ensuring that it's checked at least once for the case where gabble initiates
  the session and once where it doesn't... iotw, quite cumbersome.


  Now taking a step back and looking at how we write tests. Each test is
  basically a ``small'' scenario. To test different aspects (e.g. FT is
  openened and accepted, FT channel is opened and reject), different tests are
  written. If things are properly done, most of this is abstracted and writing
  a slighly different scenario is easy. In a lot of cases, it is done by
  copying an existing test and tweaking it slightly.

  This works reasonably well if the number of possible scenarios is quite
  small, but in practise that's not actually the case and we either write loads
  and loads of boring code or only test a few common cases (guess what happens
  in most cases)...


  When looking at the various different related scenarios, you'll see that they
  all basically exist out of mostly the same small steps, where some steps are
  slightly different to test different things. So what we actually want to do
  is to implement the various possible steps exactly once and then combine them
  into different scenarios. But writing scenarious out of little steps is
  still boring and there can be quite a few in for example jingle.

  Insted the scenarios should be generate the scenarios automatically.
  Now to make this happen you need to know how all the steps fit together (e.g.
  you can't accept a FT if there is no FT channel open). For that purpose each
  step should declare pre- and post-conditions. To give some example steps:

  def start_incoming_ft (state):
    PRE:
    POST: incoming_ft
    ....

  def accept_incoming_ft (state):
    PRE: incoming_ft && !incoming_ft_open
    POST: incoming_ft_open
    ....

  def reject_incoming_ft (state):
    PRE: incoming_ft && !incoming_ft_open
    POST:
    ....

  Possible scenarios that come out of this would be, getting an FT and
  accepting it, getting an FT and then rejecting it. But also rejecting the
  first two incoming FT's and accepting the third one. Obviously the number of
  potential scenarios goes up quite fast when there are multiple potential next
  steps, which makes it much more useful.

  Doing things this way means we can basically generate all possible scenarios
  and ensure quite exhaustive testing. A first trivial implementation would be
  to just generate all tests that do say less then 10 steps (to ensure the tests
  stop in a reasonably timeframe).

  If we go completely crazy we could combine all steps from all tests, which
  would then suddenly test if things are still ok if we're sending text
  messages while receiving a file and making an outgoing voip call. But i don't
  think we have precise enough control over gabble to actually make that
  practical (or at least, it won't be easy).

  Downside is obviously that you actually need to think how to split tests up
  into steps that are not dependant on gabbles internal state and allow for
  a large number of possible tests. But that also makes it less boring :)

  Now hopefully there are already testing frameworks that work like this (or
  that daf/wjt can implement it in 5 lines of haskell). So hopefully someone
  can point me to one those (or volunteers to implement it for tp)

  Sjoerd
-- 
We have only two things to worry about:  That things will never get
back to normal, and that they already have.