Pytest discovery fails if something is printed to stderr

Can I get any help with pytest discovery failure?

> /usr/local/bin/python ~/.vscode-server/extensions/ms-python.python-2022.0.1786462952/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear
cwd: .
[ERROR 2022-1-3 23:36:9.283]: Error discovering pytest tests:
 [r [Error]: Graceful shutdown completed in 0.000 seconds.

	at ChildProcess.<anonymous> (/root/.vscode-server/extensions/ms-python.python-2022.0.1786462952/out/client/extension.js:32:39235)
	at Object.onceWrapper (events.js:422:26)
	at ChildProcess.emit (events.js:315:20)
	at maybeClose (internal/child_process.js:1048:16)
	at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)]

Tried many different options with "python.testing.cwd" and "python.testing.pytestArgs" but without luck.

I used unit tests in the past and they work fine in VS Code.

Also I use VS Code with "Remote Container" and work inside the Docker container and run tests inside it.

Hope that you can solve this issue soon, because with pytest it doesn't work for me. And I have no clear vision why. Thanks.

Kim-Adeline Miguel (@kimadeline)

Hi Kim-Adeline Miguel (@kimadeline)

Please let me know if I can provide any useful details that may help to identify the issue. Thanks

What I may say, in my current project it's two apps in one codebase. Depending on the env variable (that is set in the .env file) - it may be run in one or another way.

When I run the tests, I do it like this APP_ENV=test APP_NAME=backend pytest backend and it runs only the tests for the part of the project. If I need to run other part of tests - I do APP_ENV=test APP_NAME=frontend pytest frontend

I did not find how make the test discovery that is used by VS Code to specify env variables.

All what I need from VS Code discovery - just to show me a list of all tests I have in the codebase. And also to have a "run" icon near each test function.

Currently I do run tests manually. I have defined in my launch.json next snippet. I change it to run individual test, if I do not need to run all tests in the file:

        {
            "name": "Run current test",
            "type": "python",
            "request": "launch",
            "module": "pytest",
            "env": {
                "APP_ENV": "test",
                "APP_NAME": "backend",
            },
            "justMyCode": false,
            "args": [
                "--no-cov",
                // Test current file
                "${file}",
                // Filter tests by name
                "-k test_some_feature_name",
            ],
            "console": "integratedTerminal",
        },

It works to me fine. Only thing that I do often - opening the launch.json file and change the -k flag each time when I need to run other test. And it's annoying.

Maybe it's a way to make Test Discovery works fine with pytest for such a project?

Getting a similar error, however it seems like it's because the environment location is messed up

> conda run -n onedevex-dev --no-capture-output python ~/.vscode-server/extensions/ms-python.python-2022.0.1786462952/pythonFiles/get_output_via_markers.py ~/.vscode-server/extensions/ms-python.python-2022.0.1786462952/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear .
cwd: .
[ERROR 2022-1-4 19:17:3.378]: Error discovering pytest tests:
 [r [Error]: 
EnvironmentLocationNotFound: Not a conda environment: /private/home/calebh/miniconda3/envs/onedevex-dev/envs/onedevex-dev


	at ChildProcess.<anonymous> (/private/home/calebh/.vscode-server/extensions/ms-python.python-2022.0.1786462952/out/client/extension.js:32:39235)
	at Object.onceWrapper (events.js:422:26)
	at ChildProcess.emit (events.js:315:20)
	at maybeClose (internal/child_process.js:1048:16)
	at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)

Perhaps I'm getting something similar which seems to stem from the discovery process trying to run my module itself with bad command line options. I'm really new to setting up a new project here so very unsure if this is my problem or similar to what others have reported here.

This bit puzzles me, why is discovery trying to run podping-hivewatcher (my project) with pytest command line options.

usage: hivewatcher [options]
hivewatcher: error: unrecognized arguments: discover pytest -- --rootdir /Users/gbishko/Documents/Python-iMac/PodcastIndex/podping-hivewatcher -s --cache-clear tests

Running pytest on the command line works fine.

> ~/Library/Caches/pypoetry/virtualenvs/podping-hivewatcher-YKYAFVom-py3.8/bin/python ~/.vscode/extensions/ms-python.python-2022.0.1786462952/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests
cwd: .
[ERROR 2022-1-5 14:29:48.180]: Error discovering pytest tests:
 [r [Error]: ============================= test session starts ==============================
platform darwin -- Python 3.8.9, pytest-5.4.3, py-1.11.0, pluggy-0.13.1
rootdir: /Users/gbishko/Documents/Python-iMac/PodcastIndex/podping-hivewatcher
usage: hivewatcher [options]
hivewatcher: error: unrecognized arguments: discover pytest -- --rootdir /Users/gbishko/Documents/Python-iMac/PodcastIndex/podping-hivewatcher -s --cache-clear tests
collected 0 items / 1 error

==================================== ERRORS ====================================
________ ERROR collecting tests/integration/test_podping_hivewatcher.py ________
tests/integration/test_podping_hivewatcher.py:2: in <module>
    from podping_hivewatcher.hivewatcher import allowed_op_id, get_allowed_accounts
src/podping_hivewatcher/hivewatcher.py:10: in <module>
    from podping_hivewatcher.config import Config
src/podping_hivewatcher/config/__init__.py:164: in <module>
    args = my_parser.parse_args()
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/argparse.py:1771: in parse_args
    self.error(msg % ' '.join(argv))
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/argparse.py:2521: in error
    self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/argparse.py:2508: in exit
    _sys.exit(status)
E   SystemExit: 2
=========================== short test summary info ============================
ERROR tests/integration/test_podping_hivewatcher.py - SystemExit: 2
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.46s ===============================

Traceback (most recent call last):
  File "/Users/gbishko/.vscode/extensions/ms-python.python-2022.0.1786462952/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/Users/gbishko/.vscode/extensions/ms-python.python-2022.0.1786462952/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/Users/gbishko/.vscode/extensions/ms-python.python-2022.0.1786462952/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

	at ChildProcess.<anonymous> (/Users/gbishko/.vscode/extensions/ms-python.python-2022.0.1786462952/out/client/extension.js:32:39235)
	at Object.onceWrapper (events.js:422:26)
	at ChildProcess.emit (events.js:315:20)
	at maybeClose (internal/child_process.js:1048:16)
	at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)]

I'm getting a very similar issue - pytest discovery not working in vscode while pytest from cmd line works fine.
From the code Python output I see:

> f:\anaconda\condabin\conda.bat run -p F:\anaconda\envs\msticpy --no-capture-output python ~\.vscode\extensions\ms-python.python-2022.1.1779589815-dev\pythonFiles\get_output_via_markers.py ~\.vscode\extensions\ms-python.python-2022.1.1779589815-dev\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests

Not sure if something in conda is able to interpret ~\.vscode\extens... as %userprofile%\.vscode\extens... (I'm running on Windows where "~" doesn't resolve to anything).

If I try to copy that command (and replace the "~" with userprofile path), it fails as follows

(I've added -v flags to conda run to try to get more info):

(msticpy) E:\src\microsoft\msticpy>f:\anaconda\condabin\conda.bat run -v -v -v -p F:\anaconda\envs\msticpy --no-capture-output python %userprofile%\.vscode\extensions\ms-python.python-2022.1.1779589815-dev\pythonFiles\get_output_via_markers.py %userprofile%\.vscode\extensions\ms-python.python-2022.1.1779589815-dev\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests 
DEBUG conda.gateways.logging:set_verbosity(231): verbosity set to 3
DEBUG conda.gateways.subprocess:subprocess_call(71): executing>> C:\WINDOWS\system32\cmd.exe /d /c F:\anaconda\envs\msticpy\.tmpzij4q0j9.bat
>>>PYTHON-EXEC-OUTPUT
usage: C:\Users\Ian\.vscode\extensions\ms-python.python-2022.1.1779589815-dev\pythonFiles\testing_tools\run_adapter.py [-h] {discover} ...
C:\Users\Ian\.vscode\extensions\ms-python.python-2022.1.1779589815-dev\pythonFiles\testing_tools\run_adapter.py: error: argument cmd: invalid choice: 'C:\\Users\\Ian\\.vscode\\extensions\\ms-python.python-2022.1.1779589815-dev\\pythonFiles\\testing_tools\\run_adapter.py' (choose from 'discover')
TRACE conda.gateways.subprocess:subprocess_call(108): $ C:\WINDOWS\system32\cmd.exe /d /c F:\anaconda\envs\msticpy\.tmpzij4q0j9.bat
==> cwd: E:\src\microsoft\msticpy <==
==> exit code: 2 <==
==> stdout <==
None
==> stderr <==
None

ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['python', 'C:\\Users\\Ian\\.vscode\\extensions\\ms-python.python-2022.1.1779589815-dev\\pythonFiles\\get_output_via_markers.py', 'C:\\Users\\Ian\\.vscode\\extensions\\ms-python.python-2022.1.1779589815-dev\\pythonFiles\\testing_tools\\run_adapter.py', 'discover', 'pytest', '--', '--rootdir', '.', '-s', '--cache-clear', 'tests']' command failed.  (See above for error)
TRACE conda.gateways.disk.delete:rm_rf(165): rm_rf F:\anaconda\envs\msticpy\.tmpzij4q0j9.bat

I also tried removing the get_output_via_markers.py component of the command - still no success. But I noticed that
it seemed to be trying to execute pytest from the conda base environment rather than the current conda env.

(msticpy) E:\src\microsoft\msticpy>f:\anaconda\condabin\conda.bat run -v -v -v -p F:\anaconda\envs\msticpy --no-capture-output python %userprofile%\.vscode\extensions\ms-python.python-2022.1.1779589815-dev\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests
DEBUG conda.gateways.logging:set_verbosity(231): verbosity set to 3
DEBUG conda.gateways.subprocess:subprocess_call(71): executing>> C:\WINDOWS\system32\cmd.exe /d /c F:\anaconda\envs\msticpy\.tmp39rluiwl.bat
equivalent command: f:\anaconda\python.exe -m pytest --collect-only --rootdir . -s --cache-clear tests
TRACE conda.gateways.subprocess:subprocess_call(108): $ C:\WINDOWS\system32\cmd.exe /d /c F:\anaconda\envs\msticpy\.tmp39rluiwl.bat
==> cwd: E:\src\microsoft\msticpy <==
==> exit code: 120 <==
==> stdout <==
None
==> stderr <==
None

ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['python', 'C:\\Users\\Ian\\.vscode\\extensions\\ms-python.python-2022.1.1779589815-dev\\pythonFiles\\testing_tools\\run_adapter.py', 'discover', 'pytest', '--', '--rootdir', '.', '-s', '--cache-clear', 'tests']' command failed.  (See above for error)
TRACE conda.gateways.disk.delete:rm_rf(165): rm_rf F:\anaconda\envs\msticpy\.tmp39rluiwl.bat

This is the line I'm referring to - although not sure if this is a anything to do with the problem - it should be f:\anaconda\envs\msticpy\python.exe

equivalent command: f:\anaconda\python.exe -m pytest --collect-only --rootdir . -s --cache-clear tests

The result is the same with or without the -v flags.

Anton Danylchenko (@1st)

Hi everyone, a couple of things here:

Issues with conda are most likely related to Debugging does not work with a conda environment #18436, since we recently changed how we run tools when in a conda environment. We hope to make a hotfix release today or tomorrow, please try it out whenever it comes out, and if it still doesn't work please open a separate issue;
When creating a conda environment, make sure you create it with the python=3.XXX argument, or the extension won't recognize it as a conda environment;
We currently do not support setting environment variables in the test runner, we have an open feature request for it: Support setting environmental variables for the test runner #17171. Please upvote it to show your interest and help us prioritize it! A workaround for now would be to specify your environment variables in a .env file.

Anton Danylchenko (@1st) and Brian of London (@brianoflondon), please provide a sample repo that I can use to try and replicate your issues.

Kim-Adeline Miguel (@kimadeline)

Kim-Adeline Miguel (@kimadeline) #19069

Mine is working again seemingly... I'll open up a new issue if it returns

Edit: ah ok, I think I've discovered the source of the error, will open a new issue.
Edit2: Opened as #19090

I struggle with a similar issue and I think it depends on a similar problem with printing to stderr.

At the moment one of my tests is failing and terminal output looks as following:

❯ pytest
================================================== test session starts ===================================================
platform linux -- Python 3.10.4, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /run/media/makanu/Daten/GIT/boxsup_pytorch/.venv/bin/python
cachedir: .pytest_cache
rootdir: /run/media/makanu/Daten/GIT/boxsup_pytorch, configfile: pyproject.toml
plugins: cov-2.12.1
collected 4 items                                                                                                        

tests/test_boxsup_pytorch.py::test_project_info PASSED                                                             [ 25%]
tests/test_losses.py::test_overlapping_1 PASSED                                                                    [ 50%]
tests/test_losses.py::test_inter_o_union_even PASSED                                                               [ 75%]
tests/test_losses.py::test_inter_o_union_uneven FAILED                                                             [100%]

======================================================== FAILURES ========================================================
_______________________________________________ test_inter_o_union_uneven ________________________________________________

bounding_box = array([[0, 1],
       [0, 0]])
full_overlap_cands = array([[[0, 1],
        [0, 0]],

       [[0, 1],
        [0, 0]],

       [[0, 1],
        [0, 0]]])

    def test_inter_o_union_uneven(bounding_box:np.array, full_overlap_cands:np.array):
        """test 1: uneven overlapping"""
>       print(inter_o_union(bounding_box, full_overlap_cands))

tests/test_losses.py:135: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

pred = array([[0, 1],
       [0, 0]])
target = array([[[0, 1],
        [0, 0]],

       [[0, 1],
        [0, 0]],

       [[0, 1],
        [0, 0]]])

    def inter_o_union(pred:np.array, target:np.array):
        """calculates Intersection over Union
    
        Args:
            pred (np.array): The prediction(s) px array
            target (np.array): The target(s) px array
    
        Returns:
            float: the divison of sum of intersection px by union px
        """
        axis = None
        if len(pred.shape) == 3 or len(target.shape) == 3: # More than 1 pred
            axis = (1,2)
    
        inter = np.sum(np.logical_and(pred, target), axis=axis)
        union = np.sum(np.logical_or(pred, target), axis=axis)
    
        print('inter: ', inter)
        print('union: ', union)
>       return union and inter / union or 0
E       ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

boxsup_pytorch/losses.py:27: ValueError
-------------------------------------------------- Captured stdout call --------------------------------------------------
inter:  [1 1 1]
union:  [1 1 1]

---------- coverage: platform linux, python 3.10.4-final-0 -----------
Name                               Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------------------
boxsup_pytorch/__init__.py             5      0      0      0   100%
boxsup_pytorch/boxsup_pytorch.py       0      0      0      0   100%
boxsup_pytorch/losses.py              12      0      2      0   100%
------------------------------------------------------------------------------
TOTAL                                 17      0      2      0   100%

Required test coverage of 100% reached. Total coverage: 100.00%
================================================ short test summary info =================================================
FAILED tests/test_losses.py::test_inter_o_union_uneven - ValueError: The truth value of an array with more than one ele...
============================================== 1 failed, 3 passed in 0.38s ===============================================

The issue happend as I started with pytest.fixtures decorators. I deleted a test which used yielding fixture. but the discover still tracked it. So deleted all caches in project directory but now the discovery fails.

The Output for python:

> ./.venv/bin/python ~/.vscode/extensions/ms-python.python-2022.6.0/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests
cwd: .
[ERROR 2022-4-6 23:20:19.95]: Error discovering pytest tests:
 [n [Error]: �[1m============================= test session starts ==============================�[0m
platform linux -- Python 3.10.4, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /run/media/makanu/Daten/GIT/boxsup_pytorch/.venv/bin/python
cachedir: .pytest_cache
rootdir: /run/media/makanu/Daten/GIT/boxsup_pytorch, configfile: pyproject.toml
plugins: cov-2.12.1
�[1mcollecting ... �[0mcollected 4 items

<Package tests>
  tests namespace.
  <Module test_boxsup_pytorch.py>
    Test module for boxsup_pytorch.
    <Function test_project_info>
      Test __author__ value.
  <Module test_losses.py>
    Test Module for losses.
    
    test scenario for overlapping_loss:
        The loss formula:
        $\Epsilon_o = {1\over N} \sum (1 - IoU(B,S)\delta (l_B, l_S)$
        
        we test with following inputs:
        
        Grid size is always 2x2
        
        test 1:
        - bounding_box: count 1, label 1
        - candidates: count 3, label 1, no overlapping
        
        expected result = 0
        
        test 2:
        - bounding_box: count 1, label 1
        - candidates: count 1, label 1, no overlapping
        - candidates: count 1, label 1, overlapping
        - candidates: count 1, label 1, full overlapping
        
        expected result = 1/2
        
        test 3:
        - bounding_box: count 1, label 1
        - candidates: count 3, label 1, full overlapping
        
        expected result = 1
        
        test 4:
        - bounding_box: count 1, label 1
        - candidates: count 3, label 2, full overlapping
        
        expected result = 0
    <Function test_overlapping_1>
      test 1: no overlapping
    <Function test_inter_o_union_even>
      test 1: even overlapping
    <Function test_inter_o_union_uneven>
      test 1: uneven overlapping


---------- coverage: platform linux, python 3.10.4-final-0 -----------
Name                               Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------------------
boxsup_pytorch/__init__.py             5      0      0      0   100%
boxsup_pytorch/boxsup_pytorch.py       0      0      0      0   100%
boxsup_pytorch/losses.py              12      9      2      0    21%   6, 18-27
------------------------------------------------------------------------------
TOTAL                                 17      9      2      0    42%

�[31m�[1mFAIL Required test coverage of 100% not reached. Total coverage: 42.11%
�[0m�[32m========================== �[32m4 tests collected�[0m�[32m in 0.41s�[0m�[32m ==========================�[0m

Traceback (most recent call last):
  File "/home/makanu/.vscode/extensions/ms-python.python-2022.6.0/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/makanu/.vscode/extensions/ms-python.python-2022.6.0/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/home/makanu/.vscode/extensions/ms-python.python-2022.6.0/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 1)

	at ChildProcess.<anonymous> (/home/makanu/.vscode/extensions/ms-python.python-2022.6.0/out/client/extension.js:2:227541)
	at Object.onceWrapper (node:events:510:26)
	at ChildProcess.emit (node:events:390:28)
	at maybeClose (node:internal/child_process:1064:16)
	at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)]

The project is completely managed by poetry no conda involved.

I have a different repo with also build with poetry and there the tests run without any problem.

I discovered the Problem but it is mind blowing:

As soon as I import numpy inside anyone of my main package modules the extension breaks. I had no issue by import numpy into my test files.

numpy.__version__ = 1.22.3

I try to downgrade numpy until it is working again and share informations.

EDIT: Update: I managed to downgrade to 1.22.0 with not running into other dependency issues. I also could install the newest release 1.21.6. For all versions I rerun pytest discovery in the test section of vscode, all with the same result.

EDIT: A work around from this post solved the issue for the moment: https://stackoverflow.com/a/62197127/10985257
Since I run coverage via pytest, which caused the issue. A custom error msg with further information about the problem might caused by coverage would be nice for easier handling in the future. In my case the --no-cov was sufficient enough to handle the problem.

I digged in this problem because I was using the environment.yml to set env variables and I discoverd the following:

the problem semms to be to me that that conda puts a message on the stderr:

https://github.com/conda/conda/blob/f356172ccbb38c7aef0a925de165c671af387657/conda/activate.py#L343-L344

Then vscode-python everytime founds a new message on the stderr, forwards it

In this case, I guess is conda's fault since it writes to the stderr this WARNING

Btw, if vscode would activate the environment before, instead of running:

conda run -n env-name --no-capture-output python ~/.vscode/extensions/ms-python.python-2022.18.2/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.18.2/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear

The problem would be fixed

I believe this is fixed on the rewrite per #18547 but please reopen if not. Thanks

Navigation Menu

Repository navigation

Pytest discovery fails if something is printed to stderr #18431

23 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Pytest discovery fails if something is printed to stderr #18431

Description

Activity

1st commented on Feb 3, 2022

1st commented on Feb 3, 2022

calebho commented on Feb 4, 2022

brianoflondon commented on Feb 5, 2022

ianhelle commented on Feb 5, 2022

kimadeline commented on Feb 8, 2022

23 remaining items

1vecera commented on May 5, 2022

pstjohn commented on May 5, 2022

MaKaNu commented on May 6, 2022

MaKaNu commented on May 7, 2022

dariocurr commented on Nov 21, 2022

eleanorjboyd commented on Sep 6, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions