Design proposal for test coverage

With the creation of the coverage API in vscode core, it is now possible to enable coverage for the python extension specifically. Below is the proposed design for how test coverage will work and includes some outstanding questions. Comments welcome as we are still in the design stage.

UX

Users can configure their coverage arguments in a similar way to how all run/debug/discovery args for testing. With the redesign of how testing arguments work with the extension of custom arguments this will be the entry point for coverage arguments. There is a new TestRunProfile kind of type coverage which will allow users to configure coverage specific configs that we will use to run coverage.

Secondly the entrypoint to run will be the buttons and other surfaces implemented in core and the default coverage config will be used unless a different one selected.

Implementation:

To implement coverage, the coverage.py library will be used. Our proposal is to not ship coverage.py and instead take the same approach as with pytest where the user must install coverage.py in their environment to use. This means that to utalize, a user will need to 1- install coverage.py, 2- create a coverage config, 3- select to run coverage.

Engineering design:

User coverage args will be sent to the python subproccess in the same means as any test args
if coverage.py is installed, and the coverage args are provided, then coverage will be calculated as described below
using the coverage.py API the python subprocess would then build a coverage payload
the payload design would include:

This would mean that coverage would support line based coverage (statement is the finite unit for coverage by vscode but python's smallest unit is line)
Due to the fact that branch coverage only lists the covered branch, and not the branches missing coverage, branch coverage will not be included at this time. If total branches can be calculated this can be reconsidered as an additional coverage stat provided.
The coverage payload will be read by the extension and displayed to the users

Other Design Concept:

The following idea was also considered: instead of using payloads for communication instead the team considered using the coverage file, like .coverage converted to XML and then read by the extension to parse the coverage and display. The pro of this would be the increase in separation between coverage and the base testing implementation that might make it easier to break out the coverage work into its own extension in the future. The con to the idea is that it will require the testing subprocess to send via information over a socket/pipe and a file thus duplicating the means of communication. It would also mean that the extension would do the heavy lifting in regards to taking the XML coverage data and converting it into the correct information design instead of doing this work on the python side. This fact would be contrary to our goal of increasing our percentage of python in the extension. ultimately we are leaning towards the first design outlined but are still open to discussion.

Given the proposal the following questions still exist:

What happens if a person has coverage.py installed, has a config of type coverage but does not include any argument to enable coverage in this config? It is required to put "" if you want to use coverage (to turn it on) but this could be confusing for users since they selected a coverage run so they might expect it to work without the adding arguments step. One option here is to just check that all coverage runs have the right args to run coverage before spinning up the subprocess.

Do we have a timeline for implementation of the same, or is the design still under discussion and not firm enough to be on roadmap?

Coverage has been implemented and is merged so you can try it now on the pre-release of the python extension!

Eleanor Boyd (@eleanorjboyd)

Eleanor Boyd (@eleanorjboyd) , Hi, I've upgraded Python extension to v2024.15.2024091301 (pre-release), but I don't see "Run with coverage" option, I see only "Run test" and "Debug test"

is your vscode version: ^1.94.0-20240913?

This one:

Version: 1.93.1 (Universal)
Commit: 38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40
Date: 2024-09-11T17:20:05.685Z
Electron: 30.4.0
ElectronBuildId: 10073054
Chromium: 124.0.6367.243
Node.js: 20.15.1
V8: 12.4.254.20-electron.0
OS: Darwin arm64 22.1.0

I've run "Check for updates", but no updates available yet.

Ah ok sorry- I didn't realize it but the coverage change requires vscode insiders due to some extra dependencies. To do so you can download insiders here and it will work from there https://code.visualstudio.com/insiders/. Otherwise this will be part of the next release- so sometime next week you will be able to access it on stable vscode

Thanks, sounds great! I'll wait for the next stable VSCode release and try it out.

Is this also functional for unittests via Django? I'm getting

...site-packages/coverage/control.py:894: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")

.vscode-server-insiders/extensions/ms-python.python-2024.15.2024092301-linux-x64/python_files/testing_tools/socket_manager.py", line 28, in connect
    self._socket.connect(self.name)
FileNotFoundError: [Errno 2] No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "vscode-server-insiders/extensions/ms-python.python-2024.15.2024092301-linux-x64/python_files/unittestadapter/execution.py", line 409, in <module>
    send_post_request(payload_cov, test_run_pipe)
  File "vscode-server-insiders/extensions/ms-python.python-2024.15.2024092301-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 348, in send_post_request
    raise VSCodeUnittestError(error_msg) from error
unittestadapter.pvsc_utils.VSCodeUnittestError: Error attempting to connect to extension named pipe /tmp/python-test-results-b7fef5ecfc1ad1eb6b3d.sock[vscode-unittest]: [Errno 2] No such file or directory
Exception ignored in atexit callback: <function execute_eot_and_cleanup at 0x7fbbdfad96c0>
Traceback (most recent call last):
  File ".vscode-server-insiders/extensions/ms-python.python-2024.15.2024092301-linux-x64/python_files/unittestadapter/execution.py", line 274, in execute_eot_and_cleanup
    send_post_request(eot_payload, test_run_pipe)
  File ".vscode-server-insiders/extensions/ms-python.python-2024.15.2024092301-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 348, in send_post_request
    raise VSCodeUnittestError(error_msg) from error
unittestadapter.pvsc_utils.VSCodeUnittestError: Error attempting to connect to extension named pipe /tmp/python-test-results-b7fef5ecfc1ad1eb6b3d.sock[vscode-unittest]: [Errno 2] No such file or directory

coverage.py is installed.

Michael D. Smith (@msmitherdc)

Michael D. Smith (@msmitherdc) I will need to investigate coverage and Django together- thank you for surfacing this!

link to issue to track Django and Coverage together: #24199

Eleanor Boyd (@eleanorjboyd)

Eleanor Boyd (@eleanorjboyd) ,
I have upgraded to VSCode 1.94:

Version: 1.94.0 (Universal)
Commit: d78a74bcdfad14d5d3b1b782f87255d802b57511
Date: 2024-10-02T13:08:12.626Z
Electron: 30.5.1
ElectronBuildId: 10262041
Chromium: 124.0.6367.243
Node.js: 20.16.0
V8: 12.4.254.20-electron.0
OS: Darwin arm64 22.1.0

But I don't see "Run with coverage" option. pytest-cov and coverage are installed.

For the future reference: I've switched to pre-release version of Python extension v2024.17.2024100202 (pre-release) and it works great!

I have switched to the pre-release version of Python extension v2024.17.2024100202 (pre-release) but still getting the error while running with coverage

UNITTEST ERROR: TEST_RUN_PIPE is not set at the time of unittest trying to send data. Please confirm this environment variable is not being changed or removed as it is required for successful test discovery and execution.TEST_RUN_PIPE = 

Exception ignored in atexit callback: <function execute_eot_and_cleanup at 0x75097171edd0>
Traceback (most recent call last):
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/execution.py", line 274, in execute_eot_and_cleanup
    send_post_request(eot_payload, test_run_pipe)
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 335, in send_post_request
    raise VSCodeUnittestError(error_msg)
unittestadapter.pvsc_utils.VSCodeUnittestError: UNITTEST ERROR: TEST_RUN_PIPE is not set at the time of unittest trying to send data. Please confirm this environment variable is not being changed or removed as it is required for successful test discovery and execution.TEST_RUN_PIPE = 


Found 1 test(s).
System check identified no issues (0 silenced).

 ebz_env/ peg_env/lib/python3.10/site-packages/coverage/control.py:894: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")
Traceback (most recent call last):
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 342, in send_post_request
    __writer.connect()
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/testing_tools/socket_manager.py", line 28, in connect
    self._socket.connect(self.name)
FileNotFoundError: [Errno 2] No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/execution.py", line 404, in <module>
    send_post_request(payload_cov, test_run_pipe)
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 346, in send_post_request
    raise VSCodeUnittestError(error_msg) from error
unittestadapter.pvsc_utils.VSCodeUnittestError: Error attempting to connect to extension named pipe /run/user/1002/python-test-results-234754ce8149ed53015a.sock[vscode-unittest]: [Errno 2] No such file or directory
Exception ignored in atexit callback: <function execute_eot_and_cleanup at 0x743d5fb2ed40>
Traceback (most recent call last):
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/execution.py", line 274, in execute_eot_and_cleanup
    send_post_request(eot_payload, test_run_pipe)
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 346, in send_post_request
    raise VSCodeUnittestError(error_msg) from error
unittestadapter.pvsc_utils.VSCodeUnittestError: Error attempting to connect to extension named pipe /run/user/1002/python-test-results-234754ce8149ed53015a.sock[vscode-unittest]: [Errno 2] No such file or directory

And you don't override any variables? This message is a sign we can't find a env var we set to use for communication:

UNITTEST ERROR: TEST_RUN_PIPE is not set at the time of unittest trying to send data. Please confirm this environment variable is not being changed or removed as it is required for successful test discovery and execution

Secondly does it work with regular run and discovery? Both of these use the same TEST_RUN_PIPE so im surprised you can get those to work and not coverage

The terminal command for coverage works fine , and also my tests are been discovered in VS code. Also, the Run Tests button component in the testing works fine, but Run Tests with coverage runs all the tests but at the end fails with above error .

I have switched to the pre-release version of Python extension v2024.17.2024100202 (pre-release) but still getting the error while running with coverage

UNITTEST ERROR: TEST_RUN_PIPE is not set at the time of unittest trying to send data. Please confirm this environment variable is not being changed or removed as it is required for successful test discovery and execution.TEST_RUN_PIPE = 

Exception ignored in atexit callback: <function execute_eot_and_cleanup at 0x75097171edd0>
Traceback (most recent call last):
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/execution.py", line 274, in execute_eot_and_cleanup
    send_post_request(eot_payload, test_run_pipe)
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 335, in send_post_request
    raise VSCodeUnittestError(error_msg)
unittestadapter.pvsc_utils.VSCodeUnittestError: UNITTEST ERROR: TEST_RUN_PIPE is not set at the time of unittest trying to send data. Please confirm this environment variable is not being changed or removed as it is required for successful test discovery and execution.TEST_RUN_PIPE = 


Found 1 test(s).
System check identified no issues (0 silenced).

 ebz_env/ peg_env/lib/python3.10/site-packages/coverage/control.py:894: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")
Traceback (most recent call last):
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 342, in send_post_request
    __writer.connect()
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/testing_tools/socket_manager.py", line 28, in connect
    self._socket.connect(self.name)
FileNotFoundError: [Errno 2] No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/execution.py", line 404, in <module>
    send_post_request(payload_cov, test_run_pipe)
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 346, in send_post_request
    raise VSCodeUnittestError(error_msg) from error
unittestadapter.pvsc_utils.VSCodeUnittestError: Error attempting to connect to extension named pipe /run/user/1002/python-test-results-234754ce8149ed53015a.sock[vscode-unittest]: [Errno 2] No such file or directory
Exception ignored in atexit callback: <function execute_eot_and_cleanup at 0x743d5fb2ed40>
Traceback (most recent call last):
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/execution.py", line 274, in execute_eot_and_cleanup
    send_post_request(eot_payload, test_run_pipe)
  File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 346, in send_post_request
    raise VSCodeUnittestError(error_msg) from error
unittestadapter.pvsc_utils.VSCodeUnittestError: Error attempting to connect to extension named pipe /run/user/1002/python-test-results-234754ce8149ed53015a.sock[vscode-unittest]: [Errno 2] No such file or directory

Now if you try in either the released version (v2024.16.0) or pre-released version (v2024.17.0) . This error is coming even when the test are sucessful

----------------------------------------------------------------------
Ran 1 test in 1.386s

OK
Destroying test database for alias 'default'...
UNITTEST ERROR: TEST_RUN_PIPE is not set at the time of unittest trying to send data. Please confirm this environment variable is not being changed or removed as it is required for successful test discovery and execution.TEST_RUN_PIPE = 

Exception ignored in atexit callback: <function execute_eot_and_cleanup at 0x7117c1e62ef0>
Traceback (most recent call last):
  File ".vscode/extensions/ms-python.python-2024.16.0-linux-x64/python_files/unittestadapter/execution.py", line 274, in execute_eot_and_cleanup
    send_post_request(eot_payload, test_run_pipe)
  File ".vscode/extensions/ms-python.python-2024.16.0-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 335, in send_post_request
    raise VSCodeUnittestError(error_msg)
unittestadapter.pvsc_utils.VSCodeUnittestError: UNITTEST ERROR: TEST_RUN_PIPE is not set at the time of unittest trying to send data. Please confirm this environment variable is not being changed or removed as it is required for successful test discovery and execution.TEST_RUN_PIPE = 


Found 1 test(s).
System check identified no issues (0 silenced).

Exception ignored in atexit callback: <function execute_eot_and_cleanup at 0x74ba1ac22e60>
Traceback (most recent call last):
  File ".vscode/extensions/ms-python.python-2024.16.0-linux-x64/python_files/unittestadapter/execution.py", line 274, in execute_eot_and_cleanup
    send_post_request(eot_payload, test_run_pipe)
  File ".vscode/extensions/ms-python.python-2024.16.0-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 346, in send_post_request
    raise VSCodeUnittestError(error_msg) from error
unittestadapter.pvsc_utils.VSCodeUnittestError: Error attempting to connect to extension named pipe /run/user/1002/python-test-results-df021158d91dd91ca8bc.sock[vscode-unittest]: [Errno 2] No such file or directory
Finished running tests!

I just tested out "Run With Coverage" on my python project. I have the sources for my app in a sub-folder so my coveragerc contains:

[run]
source = myapp
omit = myapp/tests/*

However, when I use Run With Coverage it appear to inject --cov=. which in turn overrides my source config. The result of that is that coverage picks up python all python files from the root of my project, not just limited to ./myapp.

I've not found any way around for this. Is it a bug? Or have I missed a configuration?

See this section of the docs

For finer grain control of your coverage run when using pytest, you can edit the python.testing.pytestArgs setting to include your specifications. When the pytest argument --cov exists in python.testing.pytestArgs, the Python extension will make no additional edits to coverage args, to allow your customizations to take effect. If there is no --cov argument found, the extension will add --cov=. to the pytest args prior to run to enable coverage at the workspace root.

https://code.visualstudio.com/docs/python/testing

I found that when I had --cov argument I wasn't able to debug my tests. No breakpoints would catch. That's why I was looking for separate arguments between doing a normal test run and running tests with coverage.

Not sure why the debugger wouldn't break when --cov was used though, bug?

the debugger doesn't work with coverage- it is just a general design problem as the two interfere. Understandable - hm let me think about how I could enable coverage without adding cov=. to make sure your args are not overwritten

Thomas Thomassen (@thomthom)

Thomas Thomassen (@thomthom) can you open another issue and we can move the conversation there so I can address it specifically? And if you have a suggestion on the above problem of triggering discovery with pytest-cov without overwritten exisiting args let me know, thanks!

Navigation Menu

Repository navigation

Design proposal for test coverage #22827

UX

Implementation:

Engineering design:

Other Design Concept:

Given the proposal the following questions still exist:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Design proposal for test coverage #22827

Description

UX

Implementation:

Engineering design:

Other Design Concept:

Given the proposal the following questions still exist:

Activity

amankhandelia commented on Apr 25, 2024

eleanorjboyd commented on Sep 23, 2024

AlexanderPodorov commented on Sep 23, 2024

eleanorjboyd commented on Sep 23, 2024

AlexanderPodorov commented on Sep 23, 2024

eleanorjboyd commented on Sep 23, 2024

AlexanderPodorov commented on Sep 23, 2024

msmitherdc commented on Sep 23, 2024

eleanorjboyd commented on Sep 26, 2024

eleanorjboyd commented on Sep 26, 2024

AlexanderPodorov commented on Oct 3, 2024

AlexanderPodorov commented on Oct 3, 2024

captianboy7 commented on Oct 4, 2024

eleanorjboyd commented on Oct 4, 2024

captianboy7 commented on Oct 7, 2024

captianboy7 commented on Oct 7, 2024

thomthom commented on Jan 3, 2025

eleanorjboyd commented on Jan 6, 2025

thomthom commented on Jan 7, 2025

eleanorjboyd commented on Jan 7, 2025

eleanorjboyd commented on Jan 7, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions