<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style></style></head><body lang=EN-US link=blue vlink="#954F72"><div class=WordSection1>Hi Mark, Thanks for the response!<o:p> </o:p><ol style='margin-top:0in' start=1 type=1><li class=MsoListParagraph style='margin-left:0in;mso-list:l0 level1 lfo1'>Code coverage <o:p></o:p></li></ol>We’ll try to generate the report locally for now and then investigate how to use CI for automatic report generation. Once we have some updates it would be great to have a call to discuss them. <ol style='margin-top:0in' start=2 type=1><li class=MsoListParagraph style='margin-left:0in;mso-list:l0 level1 lfo1'>Blacklist Mesa CI files <o:p></o:p></li></ol>Based on your answer, we’ll monitor such tests from our side and will create the MRs in the future. In regard to the reported ones, we’ve created the corresponding MRs. <o:p> </o:p>Please let us know if anything from our side is needed. Thanks! <o:p> </o:p>Best Regards,<o:p> </o:p>Daria Kartavtseva | Associate Manager GlobalLogic M +38 066 887 65 88 S kartavceva_darya<o:p></o:p><a href="http://www.globallogic.com/" target="_blank">www.globallogic.com</a> <a href="http://www.globallogic.com/email_disclaimer.txt" target="_blank">http://www.globallogic.com/email_disclaimer.txt</a><o:p></o:p><o:p> </o:p><div style='mso-element:para-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'>From: <a href="mailto:mark.a.janes@intel.com">Mark Janes</a> Sent: Friday, May 24, 2019 7:54 PM To: <a href="mailto:denys.kostin@globallogic.com">Denys</a>; <a href="mailto:mesa-ci-intel@lists.freedesktop.org">mesa-ci-intel@lists.freedesktop.org</a> Cc: <a href="mailto:Intel-Graphics-Dev@globallogic.com">Intel-Graphics-Dev@globallogic.com</a>; <a href="mailto:luis.strano@intel.com">Strano, Luis</a> Subject: Re: [Mesa-ci-intel] [Intel CI]. Questions related to integration codecoverage report into the CI and workflow for blacklisted cases</div><o:p> </o:p>Denys <denys.kostin@globallogic.com> writes:<o:p> </o:p>> Hello Clayton and Intel team!><o:p> </o:p>><o:p> </o:p>> We have several points that we'd like to share and discuss with you.><o:p> </o:p>> 1) Code coverage><o:p> </o:p>> Currently, we're working on a code quality improvement by measuring code > coverage and detecting the parts of the Mesa code which aren't covered > enough with the tests.><o:p> </o:p>> To start with, we've measured code coverage with the tests using only > one architecture (Kabylake). Though, there are a lot of code paths which > are architecture specific. So, to get the complete report we need to > take into account all of them as well as all possible test frameworks used.><o:p> </o:p>> The best option here is to use Intel CI for report generating since it > already runs all the available tests at all processor's architectures.><o:p> </o:p>> So, the question is whether it is possible to generate a code coverage > report using CI or maybe you have any other tools which might help with > this issue.><o:p> </o:p>><o:p> </o:p>> 2) Blacklist Mesa CI files><o:p> </o:p>> If we understand correctly there are the files with the skipped tests > (the ones which won't be run) in Blacklist Mesa CI files.><o:p> </o:p>> If so, we're wondering what is the process of moving the tests out from > the list? Can we create the corresponding MRs if we confirm the test > cases became stable and not lead to the problems anymore?><o:p> </o:p>> For now, we've tested several test cases for x32 and x64 Mesa versions > (iris) locally and they look good (they were fixed with a patch below):><o:p> </o:p>><o:p> </o:p>> # these crash on iris on m64 and fail on m32> piglit.spec.ext_image_dma_buf_import.ext_image_dma_buf_import-sample_nv12> piglit.spec.ext_image_dma_buf_import.ext_image_dma_buf_import-sample_yuv420> piglit.spec.ext_image_dma_buf_import.ext_image_dma_buf_import-sample_yvu420><o:p> </o:p>> Fix:> https://gitlab.freedesktop.org/mesa/mesa/commit/4b5e8eb3c8d709bd7c6d1a33a114bf4b002548f8><o:p> </o:p>><o:p> </o:p>> And these test cases (checked on 19.2.0-devel-21.05-810b95e02c5-DEBUG > Mesa version x32 and x64, on KBL - both tests are passing):><o:p> </o:p>> MESA_LOADER_DRIVER_OVERRIDE=iris ./point-vertex-id gl_VertexID divisor > -auto -fbo> MESA_LOADER_DRIVER_OVERRIDE=iris ./glslparsertest > /home/den/repository/piglit/tests/glslparsertest/shaders/CorrectFull.frag > pass 1.10><o:p> </o:p>><o:p> </o:p>> To summarize, our questions are:><o:p> </o:p>><o:p> </o:p>> Is it possible to generate the code coverage report by using Intel CI > and if it's already available can you please share it with us?<o:p> </o:p>It's unlikely that the CI can be used for this purpose, given ourcurrent staffing levels.<o:p> </o:p>You could generate these reports locally, using Mesa CI automation. Theconfiguration for a debian-testing installation is documented, and themesa_ci/scripts/build_local.py script will compile/run any of the targetsuites in the same manner as the CI.<o:p> </o:p>If you want to hack the automation to generate coverage reports, we canget you started.<o:p> </o:p>> What is the process of moving these tests out of the blacklist?<o:p> </o:p>We don't have a good process. We try to write up bugs for eachblacklisted test, to ensure it isn't forgotten. Ideally the test wouldbe manually re-enabled when the bug is closed. Typically, problems arefixed without resolving things in bugzilla, or the developer resolves inbugzilla without re-enabling the test.<o:p> </o:p>A robust process would involve periodically emptying the blacklists, andre-creating them based on current test behavior. Creating blacklists isa time-consuming process, because flaky tests may fail 1% of the time --generating significant CI noise for developers until the tests are fullydisabled.<o:p> </o:p>> Can we send the MRs if we confirm that the tests should be moved from > the blacklist back to the config file?<o:p> </o:p>Yes please. This will help us.<o:p> </o:p>> Looking forward to your prompt response.><o:p> </o:p>> Thanks!><o:p> </o:p>> -- > Denys Kostin | QA engineer> GlobalLogic Inc.> S evalle363> www.globallogic.com> > http://www.globallogic.com/email_disclaimer.txt><o:p> </o:p>> _______________________________________________> Mesa-ci-intel mailing list> Mesa-ci-intel@lists.freedesktop.org> https://lists.freedesktop.org/mailman/listinfo/mesa-ci-intel<o:p> </o:p></div></body></html>