Proposal of a test suite to verify the launch of elements with different element properties

[Thibault Saunier]

Hello,

We already had this idea and it is a very good one I think! :) Also I
really believe that it is a task for GstValidate[0]
(gst-validate-launcher[1] to be precise).

Implementing that in gst-validate would mean:

  * we can write those tests in python simply describing the pipelines
as a string (have a look at the GstValidateTestGenerator concept in
GstValidate)
  * we could also run scenarios[2] on all those pipelines
  * that we have the behaviour of those pipelines verified by the
monitoring infrastruture

If you have some time and are interested in looking at implementing
that in GstValidate I would be very happy to help you doing so :)

Best regards,

Thibault

[0] http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-validate/html/ch01.html
[1] http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-validate/html/command-line-tools.html#gst-validate-launcher
[2] http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-validate/html/ScenarioFileFormat.html

On Thu, Nov 27, 2014 at 10:32 AM, Vineeth <vineeth.tm at samsung.com> wrote:
>
> Right now there are test elements which check the capabilities and such of
> the elements.
> But there are no tests to check if the elements can be launched with
> different input parameters.
>
>     To quote an example, https://bugzilla.gnome.org/show_bug.cgi?id=735952,
> the element videorate was crashing when we launch with drop_only variable
> set to TRUE.
>         But there is no way to find this out until, we actually manually test the
> element with different parameters.
>     or, https://bugzilla.gnome.org/show_bug.cgi?id=735564, in the above bug,
> gdkpixbufdec element was crashing when used with imagefreeze element.
>     or, https://bugzilla.gnome.org/show_bug.cgi?id=734763, where scenechange
> element crashes without any extra parameter.
>         And i had submitted some 6 patches to fix critical errors in different
> elements.
>
>     My idea will be something like, a test plugin, which checks for all
> these possibilities.
>         Lets say, it will take an element and finds the capabilities of the element
> and checks the compatibility with other elements.
>         If elements are compatible, then it will try to decide all possible launch
> commands for the elements (first step for this will be giving static launch
> commands,
>         and dynamic detection can be added later). And for all possible launch
> commands it will check with the all the possible parameters.
>
>         This is a very complex and huge task, but might be really useful as per me.
>
> Just for example let us take, videorate element. When we inspect the
> element, we get the capabilities of the element and the properties.
> Src capability is video/x-raw, image/jpeg, image/png
> So we need to find out all the elements which have a sink capability as
> above, and link those elements with videorate element.
> Similarly for sink capability of video rate element.
> Once we identify the elements, we can check the launch capability of the
> pipeline, with different parameters for videorate.
> Now for videorate, drop-only, skip-to-first are some of the Boolean
> parameters, so we can check the pipeline with both true and false.
> There are some parameters which are of uint64 type. It will be tooo huge to
> run the case for the full range. In these cases we can test for min, max and
> few mid range values. Thus covering the worst case scenarios as well.
>
> This will help in testing all the possible worst case scenarios which is
> possible for an element.
> And one more advantage will be listing all the possible launch commands for
> the particular element, which will be very useful for new developers.
>
>
> I have done a very basic prototype of what I am meaning to do. Some elements
> which support video/raw as sink capability and fakesink can be tested by
> using gst-check-1.0 "element name"
> I am attaching the patch file of the same. It is a very crude way to doing
> things. I just did it, so that I can get it reviewed and see if it is really
> worth anything.
> Right now I am checking only if the sink property of the element is
> video/x-raw and using videotestsrc as first element and default fakesink as
> last element.
> Getting all writable element properties(Boolean and int variants for now),
> and checking with the alternates for the default values. opposite for
> Boolean, min and max for int variants.
> This will just verify if the element can launch with videotestsrc/filesrc
> and fakesink in its pipeline, for different properties.
>
> 2 major items which will improve this drastically will be
> 1) Right now I am checking by changing only one property at a time. But we
> should able to run it for all combination of available element properties.
> This will create a lot more pipelines to verify.
>
> 2) Instead of testing just with videotestsrc/fakesink, the elements should
> be tested with all kinds of possible elements, which work with the element
> being tested.
>     For this I was thinking, I should create a hash map of all the formats
> by scanning all the elements installed in a file, to be read later when
> running the check.
>         This can either be done while compiling the elements, or a separate command
> to scan it before using gst-check command.
>     This way, we need not run the scan every time we run the check for this
> element. We can just read the generated file to see the elements compatible
> with the present element.
>
> Implementing the above two will create innumerous number of possibilities of
> forming a pipeline. But it will result in a very thorough testing of the
> elements, which might help the developers a lot in identifying issues and
> stabilizing GStreamer as a whole.
>
> Please provide your inputs on the same.
>
> Sorry about the long message :)
>
> 0001-gst-check-sample-check.patch
> <http://gstreamer-devel.966125.n4.nabble.com/file/n4669706/0001-gst-check-sample-check.patch>
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Proposal-of-a-test-suite-to-verify-the-launch-of-elements-with-different-element-properties-tp4669706.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel