Proposal of a test suite to verify the launch of elements with different element properties

[Vineeth]

Right now there are test elements which check the capabilities and such of
the elements. 
But there are no tests to check if the elements can be launched with
different input parameters.

    To quote an example, https://bugzilla.gnome.org/show_bug.cgi?id=735952,
the element videorate was crashing when we launch with drop_only variable
set to TRUE. 
	But there is no way to find this out until, we actually manually test the
element with different parameters.
    or, https://bugzilla.gnome.org/show_bug.cgi?id=735564, in the above bug,
gdkpixbufdec element was crashing when used with imagefreeze element.
    or, https://bugzilla.gnome.org/show_bug.cgi?id=734763, where scenechange
element crashes without any extra parameter.
	And i had submitted some 6 patches to fix critical errors in different
elements.

    My idea will be something like, a test plugin, which checks for all
these possibilities. 
	Lets say, it will take an element and finds the capabilities of the element
and checks the compatibility with other elements. 
	If elements are compatible, then it will try to decide all possible launch
commands for the elements (first step for this will be giving static launch
commands, 
	and dynamic detection can be added later). And for all possible launch
commands it will check with the all the possible parameters.

	This is a very complex and huge task, but might be really useful as per me.

Just for example let us take, videorate element. When we inspect the
element, we get the capabilities of the element and the properties.
Src capability is video/x-raw, image/jpeg, image/png
So we need to find out all the elements which have a sink capability as
above, and link those elements with videorate element.
Similarly for sink capability of video rate element.
Once we identify the elements, we can check the launch capability of the
pipeline, with different parameters for videorate.
Now for videorate, drop-only, skip-to-first are some of the Boolean
parameters, so we can check the pipeline with both true and false.
There are some parameters which are of uint64 type. It will be tooo huge to
run the case for the full range. In these cases we can test for min, max and
few mid range values. Thus covering the worst case scenarios as well.

This will help in testing all the possible worst case scenarios which is
possible for an element.
And one more advantage will be listing all the possible launch commands for
the particular element, which will be very useful for new developers. 


I have done a very basic prototype of what I am meaning to do. Some elements
which support video/raw as sink capability and fakesink can be tested by
using gst-check-1.0 "element name"
I am attaching the patch file of the same. It is a very crude way to doing
things. I just did it, so that I can get it reviewed and see if it is really
worth anything.
Right now I am checking only if the sink property of the element is
video/x-raw and using videotestsrc as first element and default fakesink as
last element.
Getting all writable element properties(Boolean and int variants for now),
and checking with the alternates for the default values. opposite for
Boolean, min and max for int variants.
This will just verify if the element can launch with videotestsrc/filesrc
and fakesink in its pipeline, for different properties.

2 major items which will improve this drastically will be
1) Right now I am checking by changing only one property at a time. But we
should able to run it for all combination of available element properties.
This will create a lot more pipelines to verify.

2) Instead of testing just with videotestsrc/fakesink, the elements should
be tested with all kinds of possible elements, which work with the element
being tested.
    For this I was thinking, I should create a hash map of all the formats
by scanning all the elements installed in a file, to be read later when
running the check. 
	This can either be done while compiling the elements, or a separate command
to scan it before using gst-check command.
    This way, we need not run the scan every time we run the check for this
element. We can just read the generated file to see the elements compatible
with the present element.

Implementing the above two will create innumerous number of possibilities of
forming a pipeline. But it will result in a very thorough testing of the
elements, which might help the developers a lot in identifying issues and
stabilizing GStreamer as a whole.

Please provide your inputs on the same.

Sorry about the long message :)

0001-gst-check-sample-check.patch
<http://gstreamer-devel.966125.n4.nabble.com/file/n4669706/0001-gst-check-sample-check.patch>  



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Proposal-of-a-test-suite-to-verify-the-launch-of-elements-with-different-element-properties-tp4669706.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.