Hook up Codecov for coverage reporting on C code

See https://github.com/brettcannon/cpython-ci-test/blob/master/codecov.yml for an initial configuration (only differs from the default thanks to the new config for comments).

I have been running lcov regularly. gcov / lcov coverage of C code is a bit tricky for some modules. Some tests require a real terminal (ncurses), X server (tkinter, idle), root privileges (spwd) or special system configuration (yp for nis). The make coverage-lcov target ignores coverage of 3rd party code, too.

        @ # remove 3rd party modules, system headers and internal files with
        @ # debug, test or dummy functions.
        @lcov --remove $(COVERAGE_INFO) \
            '*/Modules/_blake2/impl/*' \
            '*/Modules/_ctypes/libffi*/*' \
            '*/Modules/_decimal/libmpdec/*' \
            '*/Modules/_sha3/kcp/*' \
            '*/Modules/expat/*' \
            '*/Modules/zlib/*' \
            '*/Include/*' \
            '*/Modules/xx*.c' \
            '*/Parser/listnode.c' \
            '*/Python/pyfpe.c' \
            '*/Python/pystrcmp.c' \
            '/usr/include/*' \
            '/usr/local/include/*' \
            '/usr/lib/gcc/*' \
            --output-file $(COVERAGE_INFO)

Would we run lcov as a separate coverage step from coverage.py? Or somehow run them simultaneously? I don't know what kind of slowdown we're looking at here if we had both on and if we might run afoul of Travis' free tier limitations.

On my machine it takes almost 10 minutes to run Python's test suite with a gcov instrumented build of Python. My laptop is pretty fast (8 cores i7-4900MQ CPU @ 2.80GHz, 16 GB RAM). I'm able to utilze all cores with 10 parallel test process. It's going to be hard enough to get lcov finish on Travis without coverage.py.

$ make coverage
$ time make test
...
real    9m40.226s
user    56m49.298s
sys     1m17.350s

OK, then we can look at running it as a separate test. If you want to try and get this working now then send a PR at https://github.com/brettcannon/cpython-ci-test as that's where I plan to pull the initial config files for Travis and Codecov.

I have to look into it. I'd like to use a service that can use lcov database so we can re-use out-of-tree build and lcov excludes from make coverage-lcov.

It looks like lcov is supported by Codecov: https://github.com/codecov/codecov-bash/blob/master/codecov#L822

@tiran

Did you want me to leave this open for you, @tiran to look into lcov?

@brettcannon

Please take a look at https://github.com/abalkin/cpython/tree/ci or specifically at https://codecov.io/gh/abalkin/cpython/branch/ci. @brettcannon is that the "appropriate merging" that you are looking for?

@abalkin

@abalkin unfortunately no as there's no coverage for the Lib/ directory. If you include coverage runs for both lcov and coverage.py do the results show up in Codecov as if they were run together?

Right. I thought I've seen both, but clearly my focus was on C coverage.

We are evaluating codecov's commercial offering in my firm at the moment, so I should have a working setup "soon".

add a file (#2)

Tweaked the title in hopes that someone is able to look into getting C code coverage working.

python/core-workflow#2: Add test coverage reporting for C files

Looked into this. Got it working on my fork here: https://codecov.io/gh/ammaraskar/cpython

Codecov makes this incredibly simple, they support auto merging of multiple reports, so both the python and C parts show up in the coverage report.

Some disadvantages, the coverage instrumented build takes longer to test on travis than without coverage. My build ran for ~40 minutes compared to ~20 minutes currently. This isn't too big of a concern since the coverage build is optional and does not block the travis status but its something to consider.

and here it is as part of the pull request: https://codecov.io/gh/python/cpython/tree/f2148586600ea3a2c4a04550105f32d561f100aa

Great work! Hopefully one of us will be able to review python/cpython#7773 shortly.

This is now merged!

Navigation Menu

Repository navigation

Hook up Codecov for coverage reporting on C code #2

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Hook up Codecov for coverage reporting on C code #2

Description

Activity

tiran commented on Dec 10, 2016

brettcannon commented on Dec 10, 2016

tiran commented on Dec 10, 2016

brettcannon commented on Dec 10, 2016

tiran commented on Dec 10, 2016

brettcannon commented on Dec 11, 2016

brettcannon commented on Feb 10, 2017

abalkin commented on Feb 13, 2017

brettcannon commented on Feb 13, 2017

abalkin commented on Feb 13, 2017

brettcannon commented on Apr 10, 2017

ammaraskar commented on Jun 18, 2018

ammaraskar commented on Jun 18, 2018

brettcannon commented on Jun 18, 2018

brettcannon commented on Jun 22, 2018

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions