[LDTP-Dev] Security of test cases

[Sandro Millien]

Any test script, because of being a python script, has the full
potential power of Python.
Therefore a test script is potentially very dangerous.

We have to devise a way to make it inoffensive, and diminish the
temptation for evil minded people, to pretend to contribute tests cases,
but in fact just want to run their evil code on other peoples computer.

To this end, LDTP tests cases must provide the following guaranties to
their users :
- What ever this code does, it may not access network.
- What ever this code does, it may not alter files. ( except in
explicitly specified|declared directories.
e.g. : /var/tmp/ldtp/testreports/ )




If there is an infrastructure, that enforce that :
    1) You can contribute only test cases.
    2) A test cannot access network.
    3) A test case cannot alter the file system.
       (except in predetermined directories, for it needs to report)

then I think that most evil doers may become just uninterested.



The better place to put those things would be in the python interpreter
itself.
I mean we should have the ability to invoke some  options of the python
interpreter that will make the interpreter enforce theses rules. There
may be better solutions..., it is just a first idea.



And if the test case is meant to test network operations of an AUT ? :
My first idea to solve that would be : 
There will exist a special kind of test case script specially devised
for this exception, with some additional constraints on it :
    1) It must be very small  (In order to be easy to review by a human)
    2) It will give urls to be AUT  throw a special api. This api will
perform some checks on the urls. and apply some constraints if there are
constraints that must be applied.
    3) The allure|architecture,  of this special type of test case is
predetermined.

The goal is to add all the necessary constraints to this special kind of
test case script, to make it impossible to cheat, and make this
exception a loophole in the system.



Of course I don't mean that we work on that right now, but we must keep
in mind that sooner or later, the question of "security of test cases"
will come to bite us,... as writing the test cases of his application
will become the most natural thing for a programmer.

And we will be there providing them with the framework that doesn't have
flaws...


Cheers,
Sandro Millien