[Mesa-ci-intel] [Intel CI]. Questions related to integration codecoverage report into the CI and workflow for blacklisted cases

Tue May 28 20:29:28 UTC 2019

Thanks for clarifications, Clayton.
About flaky tests Mark already warned me about this somewhere in the BZ
tickets so that's why we sent merge requests only for two "packs" of tests.
And for sure, we are carefully check tests without any comments or marked
as "flaky".
For the test which was marked as gpu hang, i checked only initial commit
(where iris was introduced) and it failed there. In latest mesa master it
became passed, so i decided that hang (and it's fix) was somewhere between.
Next time I will find commit with a fix and also provide it in the commit
message.
Have a good day!

Denys.

вт, 28 мая 2019 г., 23:17 Clayton Craft <clayton.a.craft at intel.com>:

> On Tue, May 28, 2019 at 07:42:44PM +0300, Daria Kartavtseva wrote:
>
> >2. Blacklist Mesa CI files
> >
> >Based on your answer, we’ll monitor such tests from our side and will
> create the MRs in the future.
> >In regard to the reported ones, we’ve created the corresponding MRs.
>
> Thanks, I'm testing the MRs right now and will merge soon after if it
> looks OK.
>
> One thing you may want to keep in mind is that some tests blacklisted as
> 'flaky'
> may take dozens of iterations to fail. I will usually blacklist a test that
> intermittently fails twice, even if there are many intances of it passing
> between the two failures.
>
> In the future if you happen to know which mesa commit fixes a certain test,
> include that in the MR commit message so I can make the CI 'expected pass'
> status reflect the commit that changed the status.
>
> >
> >Please let us know if anything from our side is needed. Thanks!
> >
> >
> >Best Regards,
> >
> >Daria Kartavtseva | Associate Manager
> >GlobalLogic
> >M +38 066 887 65 88  S kartavceva_darya
> >www.globallogic.com
> >
> >http://www.globallogic.com/email_disclaimer.txt
> >
> >From: Mark Janes
> >Sent: Friday, May 24, 2019 7:54 PM
> >To: Denys; mesa-ci-intel at lists.freedesktop.org
> >Cc: Intel-Graphics-Dev at globallogic.com; Strano, Luis
> >Subject: Re: [Mesa-ci-intel] [Intel CI]. Questions related to integration
> codecoverage report into the CI and workflow for blacklisted cases
> >
> >Denys <denys.kostin at globallogic.com> writes:
> >
> >> Hello Clayton and Intel team!
> >>
> >>
> >> We have several points that we'd like to share and discuss with you.
> >>
> >> 1) Code coverage
> >>
> >> Currently, we're working on a code quality improvement by measuring code
> >> coverage and detecting the parts of the Mesa code which aren't covered
> >> enough with the tests.
> >>
> >> To start with, we've measured code coverage with the tests using only
> >> one architecture (Kabylake). Though, there are a lot of code paths which
> >> are architecture specific. So, to get the complete report we need to
> >> take into account all of them as well as all possible test frameworks
> used.
> >>
> >> The best option here is to use Intel CI for report generating since it
> >> already runs all the available tests at all processor's architectures.
> >>
> >> So, the question is whether it is possible to generate a code coverage
> >> report using CI or maybe you have any other tools which might help with
> >> this issue.
> >>
> >>
> >> 2) Blacklist Mesa CI files
> >>
> >> If we understand correctly there are the files with the skipped tests
> >> (the ones which won't be run) in Blacklist Mesa CI files.
> >>
> >> If so, we're wondering what is the process of moving the tests out from
> >> the list? Can we create the corresponding MRs if we confirm the test
> >> cases became stable and not lead to the problems anymore?
> >>
> >> For now, we've tested several test cases for x32 and x64 Mesa versions
> >> (iris) locally and they look good (they were fixed with a patch below):
> >>
> >>
> >> # these crash on iris on m64 and fail on m32
> >>
> piglit.spec.ext_image_dma_buf_import.ext_image_dma_buf_import-sample_nv12
> >>
> piglit.spec.ext_image_dma_buf_import.ext_image_dma_buf_import-sample_yuv420
> >>
> piglit.spec.ext_image_dma_buf_import.ext_image_dma_buf_import-sample_yvu420
> >>
> >> Fix:
> >>
> https://gitlab.freedesktop.org/mesa/mesa/commit/4b5e8eb3c8d709bd7c6d1a33a114bf4b002548f8
> >>
> >>
> >> And these test cases (checked on 19.2.0-devel-21.05-810b95e02c5-DEBUG
> >> Mesa version x32 and x64, on KBL - both tests are passing):
> >>
> >> MESA_LOADER_DRIVER_OVERRIDE=iris ./point-vertex-id gl_VertexID divisor
> >> -auto -fbo
> >> MESA_LOADER_DRIVER_OVERRIDE=iris ./glslparsertest
> >>
> /home/den/repository/piglit/tests/glslparsertest/shaders/CorrectFull.frag
> >> pass 1.10
> >>
> >>
> >> To summarize, our questions are:
> >>
> >>
> >> Is it possible to generate the code coverage report by using Intel CI
> >> and if it's already available can you please share it with us?
> >
> >It's unlikely that the CI can be used for this purpose, given our
> >current staffing levels.
> >
> >You could generate these reports locally, using Mesa CI automation.  The
> >configuration for a debian-testing installation is documented, and the
> >mesa_ci/scripts/build_local.py script will compile/run any of the target
> >suites in the same manner as the CI.
> >
> >If you want to hack the automation to generate coverage reports, we can
> >get you started.
> >
> >> What is the process of moving these tests out of the blacklist?
> >
> >We don't have a good process.  We try to write up bugs for each
> >blacklisted test, to ensure it isn't forgotten.  Ideally the test would
> >be manually re-enabled when the bug is closed.  Typically, problems are
> >fixed without resolving things in bugzilla, or the developer resolves in
> >bugzilla without re-enabling the test.
> >
> >A robust process would involve periodically emptying the blacklists, and
> >re-creating them based on current test behavior.  Creating blacklists is
> >a time-consuming process, because flaky tests may fail 1% of the time --
> >generating significant CI noise for developers until the tests are fully
> >disabled.
> >
> >> Can we send the MRs if we confirm that the tests should be moved from
> >> the blacklist back to the config file?
> >
> >Yes please.  This will help us.
> >
> >> Looking forward to your prompt response.
> >>
> >> Thanks!
> >>
> >> --
> >> Denys Kostin | QA engineer
> >> GlobalLogic Inc.
> >> S evalle363
> >> www.globallogic.com
> >>
> >> http://www.globallogic.com/email_disclaimer.txt
> >>
> >> _______________________________________________
> >> Mesa-ci-intel mailing list
> >> Mesa-ci-intel at lists.freedesktop.org
> >> https://lists.freedesktop.org/mailman/listinfo/mesa-ci-intel
> >
>
> >_______________________________________________
> >Mesa-ci-intel mailing list
> >Mesa-ci-intel at lists.freedesktop.org
> >https://lists.freedesktop.org/mailman/listinfo/mesa-ci-intel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/mesa-ci-intel/attachments/20190528/999cd2fe/attachment.html>