"Run Test with Coverage" runs *all* tests when test execution fails

TLDR:

do not run all tests when test execution fails
also, do not run tests at all when you cannot collect the IDs

I am currently debugging a problem in my tests, most likely related to coveragepy/coveragepy#1392. I am trying the omit workaround, but currently, I am getting vscode-pytest errors such as:

test_bug.py .                                                            [100%]Error[vscode-pytest]: unable to read testIds from temp fileNo source for code: 'C:\Git\Bug\shibokensupport\signature\lib\tool.py'.

When this happens, VS Code re-starts all tests, which is not what I want at all.

Compare this snippet:

vscode-python/python_files/vscode_pytest/run_pytest_script.py

Lines 54 to 67 in e8dd8c0

    
           try: 
        
               # Read the test ids from the file, delete file, and run pytest. 
        
               ids_path = pathlib.Path(run_test_ids_pipe) 
        
               ids = ids_path.read_text(encoding="utf-8").splitlines() 
        
               try: 
        
                   ids_path.unlink() 
        
               except Exception as e: 
        
                   print("Error[vscode-pytest]: unable to delete temp file" + str(e)) 
        
               arg_array = ["-p", "vscode_pytest", *args, *ids] 
        
               print("Running pytest with args: " + str(arg_array)) 
        
               pytest.main(arg_array) 
        
           except Exception as e: 
        
               print("Error[vscode-pytest]: unable to read testIds from temp file" + str(e)) 
        
               run_pytest(args)

IMHO, this should rather be something like (untested!) - note the return in the first block and the different arg_array in the last block:

try: 
    # Read the test ids from the file, delete file, and run pytest. 
    ids_path = pathlib.Path(run_test_ids_pipe) 
    ids = ids_path.read_text(encoding="utf-8").splitlines() 
except Exception as e: 
    print("Error[vscode-pytest]: unable to read testIds from temp file: " + str(e)) 
    return
finally:
    try: 
        ids_path.unlink() 
    except Exception as e: 
        print("Error[vscode-pytest]: unable to delete temp file: " + str(e)) 

try: 
    arg_array = ["-p", "vscode_pytest", *args, *ids] 
    print("Running pytest with args: " + str(arg_array)) 
    pytest.main(arg_array) 
except Exception as e: 
    print("Error[vscode-pytest]: unable to run tests: " + str(e)) 
    arg_array = [*args, *ids] 
    print("Running pytest with args: " + str(arg_array)) 
    run_pytest(arg_array)

hoping to take a look at fixing this soon- sorry for the delay

I'm getting something similar actually, might be relevant given closeness in time of reports:
Error[vscode-pytest]: unable to read testIds from temp file[Errno 2] No such file or directory: '/tmp/test-ids-e2faf2fe418633dcef80.txt'

I'm trying to run one test but it 'fails to detect the test ID'.

Note that I'm using devcontainers, with Python + Pylance + pytest explorer installed inside.

Also, not sure if relevant: I looked at /tmp/ but can't see any files named like this. I only see files named like python-test-discovery-698debb64df061b8b350.sock

LOGS:

2024-10-31 22:11:25.650 [info] Attempting to use temp directory for test ids file, file name: test-ids-750dd803c28e00398642.txt
env vars: "TEST_RUN_PIPE":"/tmp/python-test-results-59956af0fc8ae043f0d0.sock","RUN_TEST_IDS_PIPE":"/tmp/test-ids-750dd803c28e00398642.txt"}

2024-10-31 22:11:25.658 [info] Running pytest with arguments: /root/.vscode-server/extensions/ms-python.python-2024.16.1-linux-arm64/python_files/vscode_pytest/run_pytest_script.py --rootdir=/code/myproj

2024-10-31 22:11:25.658 [info] > /usr/local/bin/python ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-arm64/python_files/vscode_pytest/run_pytest_script.py --rootdir=.

2024-10-31 22:11:25.658 [info] cwd: .

At this point I see this in my test results:

Error[vscode-pytest]: unable to read testIds from temp file[Errno 2] No such file or directory: '/tmp/test-ids-750dd803c28e00398642.txt'

Python logs don't show anything useful after that and I cancel the test runs. I also watch ls /tmp as I clicked on a test run and only saw the -result files appear. No IDs files.

hm interesting- this seems different, could you file a new issue? We use that txt file as a temp to transfer the names of the test ids to the subprocess we spin up so I would guess it might be permissions having to do with the temp directory since we aren't doing anythign specific beyond that

hm interesting- this seems different, could you file a new issue? We use that txt file as a temp to transfer the names of the test ids to the subprocess we spin up so I would guess it might be permissions having to do with the temp directory since we aren't doing anythign specific beyond that

Done, put it in #24406

bersbersbers

Hi bersbersbers, could you clarify what you mean here: "do not run all tests when test execution fails". Which test execution fails? The reading the testIds? Also could you give it another try - we have changed to using a different communication method which might help, this is only out on the pre-release of the python extension so you would need to try it there. Thanks!

Which test execution fails?

What I mean is that in the code snippet above, pytest.main(arg_array) throws an exception (in my case, it was a NoSource exception, but it can be anything). See the far end of the console output starting with test_bug.py .. The code then incorrectly prints it is "unable to read testIds from temp file", but that is a misleading error message. IMHO, reading of test IDs and running the tests should be in two independent try blocks, see my proposed code snippet above.

Also could you give it another try

Yes, will do tomorrow. But I am fairly certain the commuication method does not have anything to do with this, because it's not the reading of test IDs that fails, but the test process itself.

Also could you give it another try

I did. Obviously, since #24308 seems to be fixed, my original reproducer does not work any more. But I can still reproduce the behavior by adding

if any("::" in arg for arg in args):
    raise RuntimeError("foo")

to _pytest\config\__init__.py, main() function.

(This is supposed to simulate a pytest crash similar to before. Note that the above main() function does not catch all exceptions.)

Then run test with coverage for a single test. What then happens is:

pytest starts for one test and crashes.
pytest_code reports "unable to read testIds from temp file". This is incorrect.
pytest is re-started for all tests. This is the bug.

(This is with v2024.21.2024111501)

we fix restarting tests- I think this should fix the problem, let me know if not. Thanks!

Updated to 2024.22.0, problem still exists. Which, if I may say so, is not surprising since the buggy snippet that I pointed out in the OP (#24307 (comment)) has not changed, looking at today's main branch:

vscode-python/python_files/vscode_pytest/run_pytest_script.py

Lines 54 to 67 in e8f710a

    
           try: 
        
               # Read the test ids from the file, delete file, and run pytest. 
        
               ids_path = pathlib.Path(run_test_ids_pipe) 
        
               ids = ids_path.read_text(encoding="utf-8").splitlines() 
        
               try: 
        
                   ids_path.unlink() 
        
               except Exception as e: 
        
                   print("Error[vscode-pytest]: unable to delete temp file" + str(e)) 
        
               arg_array = ["-p", "vscode_pytest", *args, *ids] 
        
               print("Running pytest with args: " + str(arg_array)) 
        
               pytest.main(arg_array) 
        
           except Exception as e: 
        
               print("Error[vscode-pytest]: unable to read testIds from temp file" + str(e)) 
        
               run_pytest(args)

Eleanor Boyd (@eleanorjboyd)

I also experience this issue with vscode-python 2024.22.1. When I run some tests with coverage, vscode launches all the tests out of nowhere.

Running pytest with args: ['-p', 'vscode_pytest', '-n', 'auto', '--color=yes', '--ignore=templates', '--rootdir=/home/tim/Workspace/ApiFlash', '--cov=.', '/home/tim/Workspace/ApiFlash/py_test/unit/dashboard/test_plans.py::test_plans_page', '/home/tim/Workspace/ApiFlash/py_test/unit/dashboard/test_plans.py::test_display_the_current_plan', '/home/tim/Workspace/ApiFlash/py_test/unit/dashboard/test_plans.py::test_plans_switch_cost']
============================= test session starts ==============================
platform linux -- Python 3.8.20, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/tim/Workspace/ApiFlash
plugins: xdist-3.6.1, cov-5.0.0, subtests-0.13.1
created: 3/3 workers
3 workers [3 items]

...                                                                      [100%]Error[vscode-pytest]: unable to read testIds from temp fileCouldn't parse '/home/tim/Workspace/ApiFlash/templates/dashboard/base.html' as Python source: 'invalid syntax' at line 1
============================= test session starts ==============================
platform linux -- Python 3.8.20, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/tim/Workspace/ApiFlash
plugins: xdist-3.6.1, cov-5.0.0, subtests-0.13.1
collected 569 items
[...]

Vscode does that only when I run tests with coverage. If I run tests normally it does not launch all tests for no reason.
Also, if I add --cov=path_to_python_sources it works as expected.
It looks like the issue happens because vscode-python adds --cov=. by default and fails to process some non-python files.

Eleanor Boyd (@eleanorjboyd) I think this issue should be reopened because it's not fixed.

Let me know if there is more information you need.

ah yes good point- let me investigate further! Thanks

Navigation Menu

Repository navigation

"Run Test with Coverage" runs all tests when test execution fails #24307

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	try:
	# Read the test ids from the file, delete file, and run pytest.
	ids_path = pathlib.Path(run_test_ids_pipe)
	ids = ids_path.read_text(encoding="utf-8").splitlines()
	try:
	ids_path.unlink()
	except Exception as e:
	print("Error[vscode-pytest]: unable to delete temp file" + str(e))
	arg_array = ["-p", "vscode_pytest", args, ids]
	print("Running pytest with args: " + str(arg_array))
	pytest.main(arg_array)
	except Exception as e:
	print("Error[vscode-pytest]: unable to read testIds from temp file" + str(e))
	run_pytest(args)

"Run Test with Coverage" runs *all* tests when test execution fails #24307

Description

Activity

eleanorjboyd commented on Oct 30, 2024

RamiAwar commented on Oct 31, 2024

eleanorjboyd commented on Nov 5, 2024

RamiAwar commented on Nov 8, 2024

eleanorjboyd commented on Nov 14, 2024

bersbersbers commented on Nov 14, 2024

bersbersbers commented on Nov 15, 2024

eleanorjboyd commented on Dec 10, 2024

bersbersbers commented on Dec 12, 2024

TimotheeJeannin commented on Dec 29, 2024

eleanorjboyd commented on Jan 6, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

"Run Test with Coverage" runs all tests when test execution fails #24307