[Telepathy] Black-box Testing Spec.

[David Laban]

I was talking to smcv on IRC, and he said that there wasn't currently a
black-box testing framework for connection managers. I have been taking
a look at what's on offer, and I've come up with a little blueprint for
what I think it should do and how. I'm leaving it here for comments over
the weekend. (If anyone can think of a better name than tp-test, speak
now.) I plan to get the infrastructure in place on Wed and start writing
the actual tests on Thursday if I can.

Also, I happen to be in Cambridge on Monday. Who would be around to
discuss it between 2:00 and 3:00?

David








Overview
=============

This testing module is designed to help people use test-driven
development methods to quickly write reliable connection managers. It
has 5 levels of tests, to guide new developers through creating their
connection managers. (If there is already an xep-242-like definition of
what should be implemented first, I must have missed it, because it's
not in the spec anywhere.)

It is designed to be protocol-agnostic, and treats the manager as a
black-box as much as possible. It also tries to avoid the need for
configuration, and inspects existing connections where possible. If
there are two connections of the same protocol, it will also black-box
test conn1<->protocol<->conn2. If there is an echo service like Skype's
echo123 user it will test conn<->protocol<->echo.

User Interface
=================

In order to make it easy to run for existing developers:

	The testing framework should be able to be checked out from git 	
	and run with a single command.

	If the command is run with no arguments, it should inspect and
	test all connection managers running on the current session bus.

	It should be possible to include it in build-depends, and call 	
	it with simple config information to enable "make test".

In order to make it useful as a tool for developers of new CMs:

	All failure messages should include at least a hyperlink to the 	
	relevant part of the spec.

	Tests should be run in tiers, with "vital" things first, and
	[by default] other tests should not be run until these have
	passed.

	There should be a way to make it listen on the bus for your 	
	connection manager and automatically run through all tests
	whenever you restart it.

In order to make it easy to add new tests:

	There should be a set of templates from which to create new 	
	tests. This should be well documented.

	Tests added to the git repo should be automatically discovered, 	
	to avoid repetition. Protocol-specific tests local to the
	connection manager should be enabled by pointing to the
	containing folder.



Implementation
=================

The framework will be written in Python, using nose for discovery (since
I'm already familiar with this), and dbus-python + servicetest.py taken
from smcv's mission control branch (as people are already familiar with
writing tests using that, and there's something pleasingly "erlang"
about it).

Packaging will be done using python's setuptools, and it will be
possible to easy_install into a virtualenv for easy deployment. If
source-debs are required, I'll create them using
http://easy-deb.sourceforge.net/ and do a bit of hand-tuning.




Specific tests
------------------

I've started brainstorming some specific tests that should be written.
This may not be the order I write them in. If anyone has anything to
add, tell me. I have assigned each test to a tier. 1 is run first, then
2 and so on.

1: These are a requirement for getting off the ground and being
recognised by empathy etc.

2: These tests are a requirement for basic IM functionality in empathy
etc. This includes presence and text chat: things that most protocols
support. It should be very easy to pass this level, so that people don't
get discouraged early on.

3: These tests are the polish that you need to pass if you want to start
 distributing your first releases. This includes the slightly pedantic
things in the spec which you don't *need* in order to get something
working but will prevent a lot of crashes.

4: This includes the little things that make the user interface more
pleasant to use, like aliasing and avatars. It also includes file
transfers. Basically all of the stuff that Pidgin supports for most of
its protocols. This level is also the place for the *really* pedantic
regression tests.

5: This level includes all of the bonus functionality that pidgin
doesn't support, like voice/video, stream tubes and the like.

Generated tests
""""""""""""""""""""

Some tests should be generated from the spec, I really need to read more
about how tp-python is generated before I implement these.

These will all be in level 3. If any really are needed in level 2, they
should be copied in by hand, along with enough helpful error messages to
help the n00bs.

Objects implementing this interface must also implement...
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

This will be done by using Introspect() and comparing against a graph of
dependant interfaces stored as a dict [does anyone have a dot file for
this, or should I generate one myself?]


Signature checks for functions/signals/properties.
'''''''''''''''''''''''''''''''''''''''''''''''''''''

Once again, this will be done by introspection. It might also be worth
sniffing and validating all traffic while other tests are being run, but
we'll see.

ConnectionManager
""""""""""""""""""""

Testing the creation of new connections is difficult to do without
configs. I will leave this to be done by the user in Empathy/whatever
for now.

1: I will check that each connection manager has valid .manager and
.profile files associated with it, and explain where they should go if
missing.

3: I will also check for a .service file, and if missing, point out how
to reference the XDG spec when installing stuff.

Connection
""""""""""""

Connections will be searched for using a hack I wrote for cupsandstring.

3: Check that Disonnect() destroys the connection object. Then
MissionControl.SetPresence(2, '') and check that it comes back online.

3: Check that GetProtocol() conforms with the naming scheme of the object.

1: GetSelfHandle() and call InspectHandles() on it. This is required for
testing Text Channels and so on later.

1: Check that GetStatus returns CONNECTED.

1: Call ListChannels() but ignore the result. #deprecated I know, but if
it works it works. If not, try using the Requests interface method.

1: Check that either Requests.CreateChannel() or RequestChannel()
exists, but don't call it.

2: If there are multiple Connection objects, try to request a text
channel going between the two. Otherwise, try

Connection.Interface.Aliasing
'''''''''''''''''''''''''''''''

This should all be in level 3, I think.

Connection.Interface.Avatars
'''''''''''''''''''''''''''''

This should all be in level 4.

Connection.Interface.Capabilities
''''''''''''''''''''''''''''''''''''

This should all be in level 3+, I think.

Connection.Interface.Contacts
''''''''''''''''''''''''''''''''

This should all be in level 3+, I think, since this will probably be
implemented by calling other functions from the other interfaces.

Connection.Interface.Requests
'''''''''''''''''''''''''''''''''

This should either be in 2 or 3. I don't know what the recommended order
of implementation is for this vs the equivalent methods on Connection.
All I know is that it's not obvious to new developers how to implement
properties under tp-python currently, as they don't appear in the
generated code anywhere. For this reason, making people implement this
first may be off-putting.

Connection.Interface.Presence
'''''''''''''''''''''''''''''''

This should be in level 3, with the first check being for
SimplePresence. If it is missing then it should be skipped with only a
warning.

Connection.Interface.SimplePresence
'''''''''''''''''''''''''''''''''''''

This should be in 2, I think.


Channel
"""""""""

Level 2

Channel.Type.ContactList
'''''''''''''''''''''''''

Level 2 or 3. Does idle even support this?

Channel.Type.StreamedMedia
''''''''''''''''''''''''''''

level 5

Channel.Type.RoomList
''''''''''''''''''''''''

level 2 if using msn-style chats.

Channel.Type.Text
''''''''''''''''''

If a text channel is discovered which connects *only* with an account
controlled by the same connection manager, the following tests will be
performed:

Send randomly generated strings and wait for them to be received on the
other side.
3: Messages should not be received before they are sent
3: Messages should be received in order.
2: The body received should be exactly the same as that sent.

Channel.Type.StreamTube
'''''''''''''''''''''''''

Level 5

Channel.Type.DBusTube
''''''''''''''''''''''

level 5

Channel.Type.FileTransfer
'''''''''''''''''''''''''''

level 4