[Mesa-ci-intel] [Intel CI]. Questions related to integration codecoverage report into the CI and workflow for blacklisted cases

Tue May 28 16:42:44 UTC 2019

Hi Mark,

Thanks for the response!

1. Code coverage 

We’ll try to generate the report locally for now and then investigate how to use CI for automatic report generation. Once we have some updates it would be great to have a call to discuss them. 

2. Blacklist Mesa CI files

Based on your answer, we’ll monitor such tests from our side and will create the MRs in the future.
In regard to the reported ones, we’ve created the corresponding MRs. 

Please let us know if anything from our side is needed. Thanks!

Best Regards,

Daria Kartavtseva | Associate Manager
GlobalLogic
M +38 066 887 65 88  S kartavceva_darya
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

From: Mark Janes
Sent: Friday, May 24, 2019 7:54 PM
To: Denys; mesa-ci-intel at lists.freedesktop.org
Cc: Intel-Graphics-Dev at globallogic.com; Strano, Luis
Subject: Re: [Mesa-ci-intel] [Intel CI]. Questions related to integration codecoverage report into the CI and workflow for blacklisted cases

Denys <denys.kostin at globallogic.com> writes:

> Hello Clayton and Intel team!
>
>
> We have several points that we'd like to share and discuss with you.
>
> 1) Code coverage
>
> Currently, we're working on a code quality improvement by measuring code 
> coverage and detecting the parts of the Mesa code which aren't covered 
> enough with the tests.
>
> To start with, we've measured code coverage with the tests using only 
> one architecture (Kabylake). Though, there are a lot of code paths which 
> are architecture specific. So, to get the complete report we need to 
> take into account all of them as well as all possible test frameworks used.
>
> The best option here is to use Intel CI for report generating since it 
> already runs all the available tests at all processor's architectures.
>
> So, the question is whether it is possible to generate a code coverage 
> report using CI or maybe you have any other tools which might help with 
> this issue.
>
>
> 2) Blacklist Mesa CI files
>
> If we understand correctly there are the files with the skipped tests 
> (the ones which won't be run) in Blacklist Mesa CI files.
>
> If so, we're wondering what is the process of moving the tests out from 
> the list? Can we create the corresponding MRs if we confirm the test 
> cases became stable and not lead to the problems anymore?
>
> For now, we've tested several test cases for x32 and x64 Mesa versions 
> (iris) locally and they look good (they were fixed with a patch below):
>
>
> # these crash on iris on m64 and fail on m32
> piglit.spec.ext_image_dma_buf_import.ext_image_dma_buf_import-sample_nv12
> piglit.spec.ext_image_dma_buf_import.ext_image_dma_buf_import-sample_yuv420
> piglit.spec.ext_image_dma_buf_import.ext_image_dma_buf_import-sample_yvu420
>
> Fix:
> https://gitlab.freedesktop.org/mesa/mesa/commit/4b5e8eb3c8d709bd7c6d1a33a114bf4b002548f8
>
>
> And these test cases (checked on 19.2.0-devel-21.05-810b95e02c5-DEBUG 
> Mesa version x32 and x64, on KBL - both tests are passing):
>
> MESA_LOADER_DRIVER_OVERRIDE=iris ./point-vertex-id gl_VertexID divisor 
> -auto -fbo
> MESA_LOADER_DRIVER_OVERRIDE=iris ./glslparsertest 
> /home/den/repository/piglit/tests/glslparsertest/shaders/CorrectFull.frag 
> pass 1.10
>
>
> To summarize, our questions are:
>
>
> Is it possible to generate the code coverage report by using Intel CI 
> and if it's already available can you please share it with us?

It's unlikely that the CI can be used for this purpose, given our
current staffing levels.

You could generate these reports locally, using Mesa CI automation.  The
configuration for a debian-testing installation is documented, and the
mesa_ci/scripts/build_local.py script will compile/run any of the target
suites in the same manner as the CI.

If you want to hack the automation to generate coverage reports, we can
get you started.

> What is the process of moving these tests out of the blacklist?

We don't have a good process.  We try to write up bugs for each
blacklisted test, to ensure it isn't forgotten.  Ideally the test would
be manually re-enabled when the bug is closed.  Typically, problems are
fixed without resolving things in bugzilla, or the developer resolves in
bugzilla without re-enabling the test.

A robust process would involve periodically emptying the blacklists, and
re-creating them based on current test behavior.  Creating blacklists is
a time-consuming process, because flaky tests may fail 1% of the time --
generating significant CI noise for developers until the tests are fully
disabled.

> Can we send the MRs if we confirm that the tests should be moved from 
> the blacklist back to the config file?

Yes please.  This will help us.

> Looking forward to your prompt response.
>
> Thanks!
>
> -- 
> Denys Kostin | QA engineer
> GlobalLogic Inc.
> S evalle363
> www.globallogic.com
>   
> http://www.globallogic.com/email_disclaimer.txt
>
> _______________________________________________
> Mesa-ci-intel mailing list
> Mesa-ci-intel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/mesa-ci-intel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/mesa-ci-intel/attachments/20190528/10ffa152/attachment.html>