[Mesa-ci-intel] [Intel CI]. Questions related to integration codecoverage report into the CI and workflow for blacklisted cases

Tue May 28 22:38:16 UTC 2019

On Tue, May 28, 2019 at 11:29:28PM +0300, Denys Kostin wrote:
>Thanks for clarifications, Clayton.
>About flaky tests Mark already warned me about this somewhere in the BZ
>tickets so that's why we sent merge requests only for two "packs" of tests.
>And for sure, we are carefully check tests without any comments or marked
>as "flaky".
>For the test which was marked as gpu hang, i checked only initial commit
>(where iris was introduced) and it failed there. In latest mesa master it
>became passed, so i decided that hang (and it's fix) was somewhere between.
>Next time I will find commit with a fix and also provide it in the commit
>message.

Ah, if you don't know the commit, it's not critical to bisect it to find it. I
just wanted to mention that if you happened to know the commit that fixed it, it
would be nice to include it. If not, no big deal :)

>Have a good day!
>
>Denys.
>
>
>вт, 28 мая 2019 г., 23:17 Clayton Craft <clayton.a.craft at intel.com>:
>
>> On Tue, May 28, 2019 at 07:42:44PM +0300, Daria Kartavtseva wrote:
>>
>> >2. Blacklist Mesa CI files
>> >
>> >Based on your answer, we’ll monitor such tests from our side and will
>> create the MRs in the future.
>> >In regard to the reported ones, we’ve created the corresponding MRs.
>>
>> Thanks, I'm testing the MRs right now and will merge soon after if it
>> looks OK.
>>
>> One thing you may want to keep in mind is that some tests blacklisted as
>> 'flaky'
>> may take dozens of iterations to fail. I will usually blacklist a test that
>> intermittently fails twice, even if there are many intances of it passing
>> between the two failures.
>>
>> In the future if you happen to know which mesa commit fixes a certain test,
>> include that in the MR commit message so I can make the CI 'expected pass'
>> status reflect the commit that changed the status.
>>
>> >
>> >Please let us know if anything from our side is needed. Thanks!
>> >
>> >
>> >Best Regards,
>> >
>> >Daria Kartavtseva | Associate Manager
>> >GlobalLogic
>> >M +38 066 887 65 88  S kartavceva_darya
>> >www.globallogic.com
>> >
>> >http://www.globallogic.com/email_disclaimer.txt
>> >
>> >From: Mark Janes
>> >Sent: Friday, May 24, 2019 7:54 PM
>> >To: Denys; mesa-ci-intel at lists.freedesktop.org
>> >Cc: Intel-Graphics-Dev at globallogic.com; Strano, Luis
>> >Subject: Re: [Mesa-ci-intel] [Intel CI]. Questions related to integration
>> codecoverage report into the CI and workflow for blacklisted cases
>> >
>> >Denys <denys.kostin at globallogic.com> writes:
>> >
>> >> Hello Clayton and Intel team!
>> >>
>> >>
>> >> We have several points that we'd like to share and discuss with you.
>> >>
>> >> 1) Code coverage
>> >>
>> >> Currently, we're working on a code quality improvement by measuring code
>> >> coverage and detecting the parts of the Mesa code which aren't covered
>> >> enough with the tests.
>> >>
>> >> To start with, we've measured code coverage with the tests using only
>> >> one architecture (Kabylake). Though, there are a lot of code paths which
>> >> are architecture specific. So, to get the complete report we need to
>> >> take into account all of them as well as all possible test frameworks
>> used.
>> >>
>> >> The best option here is to use Intel CI for report generating since it
>> >> already runs all the available tests at all processor's architectures.
>> >>
>> >> So, the question is whether it is possible to generate a code coverage
>> >> report using CI or maybe you have any other tools which might help with
>> >> this issue.
>> >>
>> >>
>> >> 2) Blacklist Mesa CI files
>> >>
>> >> If we understand correctly there are the files with the skipped tests
>> >> (the ones which won't be run) in Blacklist Mesa CI files.
>> >>
>> >> If so, we're wondering what is the process of moving the tests out from
>> >> the list? Can we create the corresponding MRs if we confirm the test
>> >> cases became stable and not lead to the problems anymore?
>> >>
>> >> For now, we've tested several test cases for x32 and x64 Mesa versions
>> >> (iris) locally and they look good (they were fixed with a patch below):
>> >>
>> >>
>> >> # these crash on iris on m64 and fail on m32
>> >>
>> piglit.spec.ext_image_dma_buf_import.ext_image_dma_buf_import-sample_nv12
>> >>
>> piglit.spec.ext_image_dma_buf_import.ext_image_dma_buf_import-sample_yuv420
>> >>
>> piglit.spec.ext_image_dma_buf_import.ext_image_dma_buf_import-sample_yvu420
>> >>
>> >> Fix:
>> >>
>> https://gitlab.freedesktop.org/mesa/mesa/commit/4b5e8eb3c8d709bd7c6d1a33a114bf4b002548f8
>> >>
>> >>
>> >> And these test cases (checked on 19.2.0-devel-21.05-810b95e02c5-DEBUG
>> >> Mesa version x32 and x64, on KBL - both tests are passing):
>> >>
>> >> MESA_LOADER_DRIVER_OVERRIDE=iris ./point-vertex-id gl_VertexID divisor
>> >> -auto -fbo
>> >> MESA_LOADER_DRIVER_OVERRIDE=iris ./glslparsertest
>> >>
>> /home/den/repository/piglit/tests/glslparsertest/shaders/CorrectFull.frag
>> >> pass 1.10
>> >>
>> >>
>> >> To summarize, our questions are:
>> >>
>> >>
>> >> Is it possible to generate the code coverage report by using Intel CI
>> >> and if it's already available can you please share it with us?
>> >
>> >It's unlikely that the CI can be used for this purpose, given our
>> >current staffing levels.
>> >
>> >You could generate these reports locally, using Mesa CI automation.  The
>> >configuration for a debian-testing installation is documented, and the
>> >mesa_ci/scripts/build_local.py script will compile/run any of the target
>> >suites in the same manner as the CI.
>> >
>> >If you want to hack the automation to generate coverage reports, we can
>> >get you started.
>> >
>> >> What is the process of moving these tests out of the blacklist?
>> >
>> >We don't have a good process.  We try to write up bugs for each
>> >blacklisted test, to ensure it isn't forgotten.  Ideally the test would
>> >be manually re-enabled when the bug is closed.  Typically, problems are
>> >fixed without resolving things in bugzilla, or the developer resolves in
>> >bugzilla without re-enabling the test.
>> >
>> >A robust process would involve periodically emptying the blacklists, and
>> >re-creating them based on current test behavior.  Creating blacklists is
>> >a time-consuming process, because flaky tests may fail 1% of the time --
>> >generating significant CI noise for developers until the tests are fully
>> >disabled.
>> >
>> >> Can we send the MRs if we confirm that the tests should be moved from
>> >> the blacklist back to the config file?
>> >
>> >Yes please.  This will help us.
>> >
>> >> Looking forward to your prompt response.
>> >>
>> >> Thanks!
>> >>
>> >> --
>> >> Denys Kostin | QA engineer
>> >> GlobalLogic Inc.
>> >> S evalle363
>> >> www.globallogic.com
>> >>
>> >> http://www.globallogic.com/email_disclaimer.txt
>> >>
>> >> _______________________________________________
>> >> Mesa-ci-intel mailing list
>> >> Mesa-ci-intel at lists.freedesktop.org
>> >> https://lists.freedesktop.org/mailman/listinfo/mesa-ci-intel
>> >
>>
>> >_______________________________________________
>> >Mesa-ci-intel mailing list
>> >Mesa-ci-intel at lists.freedesktop.org
>> >https://lists.freedesktop.org/mailman/listinfo/mesa-ci-intel
>>
>>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/mesa-ci-intel/attachments/20190528/cb083725/attachment-0001.sig>