[Bug 754052] validate: launcher: Add support to skip already passed tests

[GStreamer (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=754052

--- Comment #7 from Vineeth <vineeth.tm at samsung.com> ---
(In reply to Thibault Saunier from comment #5)
> Review of attachment 309939 [details] [review]:
> 
> The way of doing it is not really nice, could you rather do --skip-passed
> previous_result.xml (passing an Xunit file) and use the information about
> whether the test passed or not from it? It would be much cleaner :)

i tried with this method. One major issue with this method is.
First time we run the tests, we create previous_results.xml
let us say out of 100 cases there are 60 pass and 40 fail cases.
Now for 2nd time, i made some fixes for some of the failed cases.
So when i run the tests, i check the xml file and see 60 pass results and hence
skip the same and run only 40 fail cases.
Now out of the 40 fail cases, 30 cases are success and 10 cases are failure.
So the new previous_results.xml will have 30 pass cases and 10 fail cases.

Now for the 3rd time, i make some more fixes for the 10 fail cases.
So when i run the tests and check the xml file, now there are only 30 pass
cases. And i will skip only those 30 test cases.
This means the first 60 pass cases found during the first time will be run
again this time. So instead of running only 10 fail cases, we will run 60 pass
from first time and 10 fail from 2nd time...

Not sure if i explained it properly :)...

one solution for this method can be,
While running the test for the 2nd time, when we skip the 60 passed tests, even
though we dont run these passed tests, we copy the results of the passed tests
to the newly generated xunit.xml file, so that after running the tests for 2nd
time, the xml file will have 90 passed tests and 10 failed tests.

Please guide me if this is the right way, or you have any other better solution 
:)



(In reply to Thibault Saunier from comment #6)
> Any way we put your test suite in the open in
> gst-validate-integration-testsuites? :)

Very sorry about this.. I wont be able to share the media files due to company
policies :(..

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.