[Intel-gfx] BKM - Using JIRA Bugs for traceability

[Tahvanainen, Jari]

Hello folks,

Procedure FYI for the persons who are not executing tests in Jira.

GFX QA (and whoever does execution in Jira) shall follow the Rule: One must have traceability bugs (from Jira to fdo) for all the tests having FAIL or BLOCKED status. Note that on some cases one might not have the bug in fdo (e.g. security reasons), and on those cases all said below does not apply.

Note the following procedure when executing tests within Jira and having FAIL or BLOCKED verdict for the test

-          If there is failure on test case execution done in Jira (e.g. https://jira01.devtools.intel.com/secure/enav/#/57623) then one SHALL have bug as record for it. This was done correctly in this case by having defect attached to execution result - see

*         Defects: VIZ-13164<https://jira01.devtools.intel.com/browse/VIZ-13164> --> fdo.104464<https://bugs.freedesktop.org/show_bug.cgi?id=104464>

-          In Jira side bug, in order traceability Feature-Test-Result to work correctly, one shall have Traceability in Labels (was not done for VIZ-13164, I fixed this). In addition to this title should include fdo.fdoID (as fdo.104464, was correctly done in this example),  priority should be set according the bug in fdo (was not done for VIZ-13164, I fixed this).

If the above rules are followed then we have enough information in title and as links for first glance, and also way to

-          Not to show this traceability bug on Jira queries - e.g. https://jira01.devtools.intel.com/issues/?jql=issuetype%20%3D%20Bug%20and%20%22Platform%2Fs%22%20in%20(%22Coffee%20Lake%22)%20and%20status%20not%20in%20(Closed%2C%20Resolved)%20AND%20(labels%20is%20empty%20OR%20labels%20not%20in%20(Traceability))%20order%20by%20priority%20desc%2C%20component%20asc

o   One should track all these bugs on fdo, and then GFX QA should follow up traceability bugs in Jira every now and then and e.g. close the ones in Jira which are marked as fixed in fdo.

?  BTW the query of CFL traceability bugs is https://jira01.devtools.intel.com/issues/?jql=issuetype%20%3D%20Bug%20and%20%22Platform%2Fs%22%20in%20(%22Coffee%20Lake%22)%20and%20status%20not%20in%20(Closed%2C%20Resolved)%20AND%20labels%20in%20(Traceability)%20order%20by%20priority%20desc%2C%20component%20asc

-          Have full traceability for test case failure in Jira to fdo for reporting and retesting when fdo bug is marked as fixed.

o   Reporting example (still PoC tool/script)

?  CFL PV  System Testing summary:  1  FAIL,  6  UNEXECUTED,  0  BLOCKED,  91 PASS.  93 % RUN RATE,  92 % PASS RATE

*         NOTE: some of the 73 tests were executed more than once in different environments giving total 98 executed

?  2DZephyr : 0  FAIL,  0  UNEXECUTED,  0  BLOCKED,  26 PASS.  100 % RUN RATE,  100 % PASS RATE

?  3DZephyr : 1  FAIL,  0  UNEXECUTED,  0  BLOCKED,  9 PASS.  100 % RUN RATE,  90 % PASS RATE

*         BUGS: VIZ-13164/[CFL][ TRACEABILITY] fdo.104464 -Poor frame rate on 3D games

?  AppsZephyr : 0  FAIL,  0  UNEXECUTED,  0  BLOCKED,  2 PASS.  100 % RUN RATE,  100 % PASS RATE

?  CTSZephyr : 0  FAIL,  3  UNEXECUTED,  0  BLOCKED,  0 PASS.  0 % RUN RATE,  0 % PASS RATE

?  DEQZephyr : 0  FAIL,  2  UNEXECUTED,  0  BLOCKED,  0 PASS.  0 % RUN RATE,  0 % PASS RATE

?  DmcZephyr : 0  FAIL,  0  UNEXECUTED,  0  BLOCKED,  18 PASS.  100 % RUN RATE,  100 % PASS RATE

?  GucZephyr : 0  FAIL,  1  UNEXECUTED,  0  BLOCKED,  22 PASS.  95 % RUN RATE,  95 % PASS RATE

?  HucZephyr : 0  FAIL,  0  UNEXECUTED,  0  BLOCKED,  4 PASS.  100 % RUN RATE,  100 % PASS RATE

?  Kernel - display : 0  FAIL,  0  UNEXECUTED,  0  BLOCKED,  9 PASS.  100 % RUN RATE,  100 % PASS RATE

?  KernelZephyr : 0  FAIL,  0  UNEXECUTED,  0  BLOCKED,  1 PASS.  100 % RUN RATE,  100 % PASS RATE

BR. Jari

1.    PS. I also fixed (=added the Traceability on Labels for) VIZ-12145<https://jira01.devtools.intel.com/browse/VIZ-12145>, VIZ-12142<https://jira01.devtools.intel.com/browse/VIZ-12142> ...

   Jari Tahvanainen
   ---------------------------------------------------------------------
   Intel Finland Oy
   BIG 0357606-4
   Westendinkatu 7, 02160 Espoo

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/intel-gfx/attachments/20180110/05330541/attachment-0001.html>