[igt-dev] [PATCH v2 07/12] docs/code_coverage.md: document the code coverage filter script
Mauro Carvalho Chehab
mauro.chehab at linux.intel.com
Tue Apr 12 08:59:06 UTC 2022
From: Mauro Carvalho Chehab <mchehab at kernel.org>
Add documentation bits for the code_coverage_parse_info.
While here, also drop the extensions from the scripts that were renamed.
Reviewed-by: Ch Sai Gowtham <sai.gowtham.ch at intel.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab at kernel.org>
---
To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover.
See [PATCH v2 00/12] at: https://lore.kernel.org/all/cover.1649753814.git.mchehab@kernel.org/
docs/code_coverage.md | 238 +++++++++++++++++++++++++++++++++++++++---
1 file changed, 224 insertions(+), 14 deletions(-)
diff --git a/docs/code_coverage.md b/docs/code_coverage.md
index 34d56d283a75..915e800ed68b 100644
--- a/docs/code_coverage.md
+++ b/docs/code_coverage.md
@@ -132,15 +132,15 @@ Kernel that was built at the same machine, at the directory `~/linux`,
and wants to capture one file per test, it would use:
```
-./scripts/run-tests.sh -T my.testlist -k ~/linux -c scripts/code_cov_capture.sh -P
+./scripts/run-tests.sh -T my.testlist -k ~/linux -c code_cov_capture -P
```
### Code Coverage Collect script
While any script could in thesis be used, currently, there are two ones
-under `scripts/`:
+under the IGT's `scripts/` source directory:
-- `scripts/code_cov_capture.sh`:
+- `code_cov_capture`:
Assumes that the Kernel was built at the same machine, and uses
the lcov tool to generate GCC-independent code coverage data,
@@ -150,7 +150,7 @@ under `scripts/`:
Such script requires `lcov` tool to be installed at the test machine.
-- `scripts/code_cov_gather_on_test.py`:
+- `code_cov_gather_on_test`:
Generates a gzipped tarbal with the code coverage counters in
binary format. Such kind of output should then be parsed at
@@ -162,12 +162,12 @@ For each script, the igt_runner passes just one parameter: the results
directory + the test name.
For instance, if it is needed to run a test called
-`debugfs_test (read_all_entries)` using `scripts/code_cov_capture.sh`
+`debugfs_test (read_all_entries)` using `code_cov_capture`
parameter, e. g.:
```
$ echo "igt at debugfs_test@read_all_entries" > my.testlist
-$ ./scripts/run-tests.sh -T my.testlist -k ~/linux -c scripts/code_cov_capture.sh -P
+$ ./scripts/run-tests.sh -T my.testlist -k ~/linux -c code_cov_capture -P
Found test list: "/basedir/igt/build/tests/test-list.txt"
[31410.499969] [1/1] debugfs_test (read_all_entries)
[31411.060446] Storing code coverage results...
@@ -178,7 +178,7 @@ Done.
The script will be called as:
```
-scripts/code_cov_capture.sh /basedir/igt/results/code_cov/debugfs_test_read_all_entries
+code_cov_capture results/code_cov/debugfs_test_read_all_entries
```
Please notice that any character that it is not a number nor a letter at the
@@ -190,6 +190,73 @@ as titles at the lcov files.
If any extra global parameters are needed by the script, those can be sent
via shell's environment var.
+## Parsing data from code coverage *.info files
+
+The `*.info` files generated by `lcov` are plain text files that list the
+tests that were executed in runtime.
+
+The `code_cov_parse_info` script has some logic on it that allows
+printing the called functions stored inside the `*.info` file. It can also
+optionally apply the following filters. Its main options are:
+
+- `--stat` or `--statistics`
+
+ Prints code coverage statistics.
+
+ It displays function, line, branch and file coverage percentage.
+
+ The statistics report is affected by the applied filters.
+
+- `--print-coverage`, `--print` or `-p`
+
+
+ Prints the functions that were executed in runtime and how many times
+ they were reached.
+
+ The function coverage report is affected by the applied filters.
+
+- `--print-unused` or `-u`
+
+ Prints the functions that were never reached.
+
+ The function coverage report is affected by the applied filters.
+
+- `--show-lines` or `--show_lines`
+
+ When printing per-function code coverage data, always output the source
+ file and the line number where the function is defined.
+
+- `--output` *output file* or `-o` *output file*
+
+ Produces an output file merging all input files.
+
+ The generated output file is affected by the applied filters.
+
+- `--show-files` or `--show_files`
+
+ Shows the list of files that were useed to produce the code coverage
+ results.
+
+- It also has a set of parameters that filters the code coverage results:
+ `--only-drm`, `--only-i915`, `--func-filters`, `--source-filters`,
+ `--ignore-unused`.
+ When used, all coverage displayed reports, and the stored output file
+ will be affected by such filters.
+
+More details can be seen by calling:
+
+```
+code_cov_parse_info --help
+
+```
+
+or:
+
+```
+code_cov_parse_info --man
+
+```
+
### The `*.info` file format
The `*.info` files contain several fields on it, grouped into records.
@@ -224,7 +291,7 @@ So, the above example means that, inside
## Generating code coverage documentation
The `lcov` package contains the needed tools to parse and generate code
-coverage documentation. It is used by `scripts/code_cov_capture.sh` script
+coverage documentation. It is used by `code_cov_capture` script
to convery from compiler-dependent `*.gcno` counters into a
compiler-independent format (`*.info`).
@@ -243,11 +310,16 @@ source files. Some optional arguments can be used at the command line, or
can be stored at `/etc/lcovrc` or `~/.lcovrc` files.
As generating the documentation depends wheather the results were generated
-as with a single or multiple `*.info` files by `scripts/code_cov_capture.sh`
+as with a single or multiple `*.info` files by `code_cov_capture`
or stored in raw formats inside `*.tar.gz` file(s) by
-`scripts/code_cov_gather_on_test.py`, there's a script that does all the
+`code_cov_gather_on_test`, there's a script that does all the
required steps to build the code coverage html reports:
-`scripts/code_cov_gen_report.sh`.
+`code_cov_gen_report`. Besides its own command line arguments, it
+also accepts arguments to be passed to `code_cov_parse_info`.
+
+If a `code_cov_parse_info` command line parameter is passed, it will
+also call the script, in order to use a filtered `*.info` file to be
+used when generating the HTML reports.
It requires the following arguments:
@@ -276,18 +348,156 @@ It requires the following arguments:
The files specified by `--read` are gzipped tarballs containing all
`*.gcno` files and all `*.gcda` softlinks from the `/sys/kernel/debug/gcov/`
- directory at the test machine, created by
- `scripts/code_cov_gather_on_test.py` script.
+ directory at the test machine, created by `code_cov_gather_on_test` script.
- `--force-override`
Allow using a non-empty directory for `--output-dir`.
+- It also accepts `--print`, `--only-drm`, `--only-i915` and `--ignore-unused`
+ options from `code_cov_parse_info`.
+
`--info` and `--tar` are mutually exclusive and at least one of them should
be specified.
+## Code coverage capture script example
+
+### Capture, parse and generate code coverage html data
+
+The script below provides a simple yet powerful script using code
+coverage capture on a test machine that also contains the Linux
+Kernel source and objects. It assumes that LGT was installed.
+
+```
+#/bin/bash -e
+
+TESTLIST="my_tests.testlist"
+OUT_DIR="${HOME}/results"
+
+mkdir -p $OUT_DIR/html
+
+echo "igt at debugfs_test@read_all_entries" > $TESTLIST
+echo "igt at core_auth@basic-auth" >> $TESTLIST
+echo "igt at gem_exec_basic@basic" >> $TESTLIST
+
+sudo IGT_KERNEL_TREE="${HOME}/linux" igt_runner -s -o --coverage-per-test \
+ --collect-script code_cov_capture --test-list $TESTLIST \
+ /usr/local/libexec/igt-gpu-tools $OUT_DIR/ | sed s,$HOME/,,
+
+sudo chown -R $(id -u):$(id -g) $OUT_DIR/
+
+for i in $OUT_DIR/code_cov/*.info; do
+ echo -e "\n$(basename $i):"
+ code_cov_parse_info --only-drm --ignore-unused --stat $i
+done
+echo -e "\nTOTAL:"
+code_cov_parse_info --only-drm --stat --output $OUT_DIR/results.info \
+ $OUT_DIR/code_cov/*.info
+
+cd $OUT_DIR/html
+genhtml -q -s --legend --branch-coverage $OUT_DIR/results.info
+```
+
+Running such script produces the following output:
+
+```
+[3622.993304] [1/3] debugfs_test (read_all_entries)
+[3631.95] Code coverage wrote to results/code_cov/debugfs_test_read_all_entries.info
+[3626.217016] Storing code coverage results...
+[3631.957998] [2/3] core_auth (basic-auth)
+[3638.03] Code coverage wrote to results/code_cov/core_auth_basic_auth.info
+[3632.116024] Storing code coverage results...
+[3638.070869] [3/3] gem_exec_basic (basic)
+[3644.24] Code coverage wrote to results/code_cov/gem_exec_basic_basic.info
+[3638.366790] Storing code coverage results...
+Done.
+
+core_auth_basic_auth.info:
+ lines......: 11.7% (8217 of 70257 lines)
+ functions..: 7.1% (776 of 10971 functions)
+ branches...: 7.0% (3596 of 51041 branches)
+Ignored......: non-drm headers and source files where none of its code ran.
+Source files.: 23.27% (165 of 709 total), 29.57% (165 of 558 filtered)
+
+debugfs_test_read_all_entries.info:
+ lines......: 19.3% (20266 of 104802 lines)
+ functions..: 17.5% (1922 of 10971 functions)
+ branches...: 12.7% (9462 of 74555 branches)
+Ignored......: non-drm headers and source files where none of its code ran.
+Source files.: 34.70% (246 of 709 total), 44.09% (246 of 558 filtered)
+
+gem_exec_basic_basic.info:
+ lines......: 17.1% (14964 of 87503 lines)
+ functions..: 13.0% (1422 of 10971 functions)
+ branches...: 10.1% (6446 of 63758 branches)
+Ignored......: non-drm headers and source files where none of its code ran.
+Source files.: 30.89% (219 of 709 total), 39.25% (219 of 558 filtered)
+
+TOTAL:
+ lines......: 15.5% (25821 of 166849 lines)
+ functions..: 22.1% (2429 of 10971 functions)
+ branches...: 10.5% (11869 of 112665 branches)
+Ignored......: non-drm headers.
+Source files.: 78.70% (558 of 709 total)
+```
+
+### Reporting detailed function coverage stored on *.info files
+
+The `code_cov_parse_info` script can be used alone in order to provide
+a text file output containing code coverage data obtained from a *.info
+file. For example, listing code coverage usage for all functions whose name
+contains "edid_" can be done with:
+
+```
+$ echo edid_ >filter.txt
+$ code_cov_parse_info --func-filters filter.txt results/results.info -p -u --stat
+TEST: Code_coverage_tests
+__drm_get_edid_firmware_path(): unused
+__drm_set_edid_firmware_path(): unused
+displayid_iter_edid_begin(): executed 10 times
+drm_add_edid_modes(): executed 2 times
+drm_add_override_edid_modes(): unused
+drm_connector_attach_edid_property(): unused
+drm_connector_update_edid_property(): executed 8 times
+drm_dp_send_real_edid_checksum(): unused
+drm_edid_are_equal(): executed 4 times
+drm_edid_block_valid(): executed 8 times
+drm_edid_duplicate(): unused
+drm_edid_get_monitor_name(): unused
+drm_edid_header_is_valid(): executed 4 times
+drm_edid_is_valid(): executed 2 times
+drm_edid_to_eld(): executed 2 times
+drm_edid_to_sad(): unused
+drm_edid_to_speaker_allocation(): unused
+drm_find_edid_extension(): executed 22 times
+drm_get_edid_switcheroo(): unused
+drm_load_edid_firmware(): executed 2 times
+edid_firmware_get(): unused
+edid_firmware_set(): unused
+edid_fixup_preferred(): unused
+edid_get_quirks(): executed 6 times
+edid_load(): unused
+edid_open(): executed 4 times
+edid_show() from linux/drivers/gpu/drm/drm_debugfs.c: executed 4 times
+edid_show() from linux/drivers/gpu/drm/drm_sysfs.c: unused
+edid_vendor(): executed 348 times
+edid_write(): unused
+intel_panel_edid_downclock_mode(): unused
+intel_panel_edid_fixed_mode(): unused
+is_edid_digital_input_dp(): unused
+ lines......: 5.5% (5 of 91 lines)
+ functions..: 42.4% (14 of 33 functions)
+ branches...: 1.9% (1 of 52 branches)
+Ignored......: unmatched functions m/(?^:edid_)/ and source files where none of its code ran.
+Source files.: 0.90% (5 of 558 total), 55.56% (5 of 9 filtered)
+```
+
+When the function is unique, it will just display the function name and how
+many times the IGT test(s) executed it. When the same function name exists
+on multiple files (like the `edid_show()` on the above example), it will
+display multiple lines, one for each different function/file combination.
+
## References
More information is available at Kernel gcov documentation:
[Using gcov with the Linux kernel](https://www.kernel.org/doc/html/latest/dev-tools/gcov.html).
-
--
2.35.1
More information about the igt-dev
mailing list