Skip to content

Stderr output causes test discovery to show as failed #18547

Closed
Closed
Stderr output causes test discovery to show as failed#18547
Assignees
eleanorjboyd
Labels
area-testingbugIssue identified by VS Code Team member as probable bug

Description

@Cnoor0171
Cnoor0171
opened on Feb 19, 2022

Issue Type: Bug

Behaviour

Expected vs. Actual

This is duplicate of #7095, but that issue was closed because the author didn't provide steps to reproduce. The bug is still there.

Expected: the extension to be able to detect tests when pytest test discovery succeeds, even if there is output on stderr.

Actual: If there is leftover output on stderr, the extension fails to discover tests. Note that the print needs to happen during test discovery, NOT inside the test. Also note that the discovery only fails if content is written to stderr or stdout after pytest thinks it's done reporting the test collection results. For that to happen we need to spin up a separate thread that waits to print something. In my particular case, the culprit was a library that spawns a background thread for log processing that was logging a warning. Running pytest --collect-only exits cleanly, but the extension is not able to process the results.

The extension logs show:

> ./.venv/bin/python ~/.vscode-server/extensions/ms-python.python-2022.0.1814523869/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear .
cwd: .
[ERROR 2022-1-19 1:15:57.156]: Error discovering pytest tests:
 [r [Error]: something something something
	at ChildProcess.<anonymous> (/home/cnoor0171/.vscode-server/extensions/ms-python.python-2022.0.1814523869/out/client/extension.js:32:39235)
	at Object.onceWrapper (events.js:422:26)
	at ChildProcess.emit (events.js:315:20)
	at maybeClose (internal/child_process.js:1048:16)
	at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)]

Steps to reproduce:

  1. In an empty workspace, install pytest in a venv
  2. Create a file test_something.py with content
from threading import Thread
from time import sleep
import sys

def threaded_function():
    sleep(3)
    sys.stderr.write("something something something")

thread = Thread(target=threaded_function)
thread.start()

def test_passing():
    assert 1 == 1
  1. Try to run pytest discovery on workspace

Diagnostic data

  • Python version (& distribution if applicable, e.g. Anaconda): 3.9.5
  • Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
  • Value of the python.languageServer setting: Pylance
User Settings 


defaultLS: {"defaultLSType":"Pylance"}

downloadLanguageServer: true

envFile: "<placeholder>"

venvPath: "<placeholder>"

venvFolders: "<placeholder>"

condaPath: "<placeholder>"

pipenvPath: "<placeholder>"

poetryPath: "<placeholder>"

languageServer: "Pylance"

linting
• enabled: true
• cwd: "<placeholder>"
• Flake8Args: "<placeholder>"
• flake8Enabled: false
• flake8Path: "<placeholder>"
• lintOnSave: true
• banditArgs: "<placeholder>"
• banditEnabled: false
• banditPath: "<placeholder>"
• mypyArgs: "<placeholder>"
• mypyEnabled: false
• mypyPath: "<placeholder>"
• pycodestyleArgs: "<placeholder>"
• pycodestyleEnabled: false
• pycodestylePath: "<placeholder>"
• prospectorArgs: "<placeholder>"
• prospectorEnabled: false
• prospectorPath: "<placeholder>"
• pydocstyleArgs: "<placeholder>"
• pydocstyleEnabled: false
• pydocstylePath: "<placeholder>"
• pylamaArgs: "<placeholder>"
• pylamaEnabled: false
• pylamaPath: "<placeholder>"
• pylintArgs: "<placeholder>"
• pylintPath: "<placeholder>"

sortImports
• args: "<placeholder>"
• path: "<placeholder>"

formatting
• autopep8Args: "<placeholder>"
• autopep8Path: "<placeholder>"
• provider: "autopep8"
• blackArgs: "<placeholder>"
• blackPath: "<placeholder>"
• yapfArgs: "<placeholder>"
• yapfPath: "<placeholder>"

testing
• cwd: "<placeholder>"
• debugPort: 3000
• nosetestArgs: "<placeholder>"
• nosetestsEnabled: undefined
• nosetestPath: "<placeholder>"
• promptToConfigure: true
• pytestArgs: "<placeholder>"
• pytestEnabled: true
• pytestPath: "<placeholder>"
• unittestArgs: "<placeholder>"
• unittestEnabled: false
• autoTestDiscoverOnSaveEnabled: true

terminal
• activateEnvironment: true
• executeInFileDir: "<placeholder>"
• launchArgs: "<placeholder>"

experiments
• enabled: true
• optInto: []
• optOutFrom: []

insidersChannel: "off"

tensorBoard
• logDirectory: "<placeholder>"

Extension version: 2022.0.1814523869
VS Code version: Code 1.64.2 (f80445acd5a3dadef24aa209168452a3d97cc326, 2022-02-09T22:02:28.252Z)
OS version: Windows_NT x64 10.0.19044
Restricted Mode: No
Remote OS version: Linux x64 5.10.16.3-microsoft-standard-WSL2
Remote OS version: Linux x64 5.10.16.3-microsoft-standard-WSL2

System Info
Item Value
CPUs AMD Ryzen 5 2600 Six-Core Processor (12 x 3394)
GPU Status 2d_canvas: enabled
gpu_compositing: enabled
multiple_raster_threads: enabled_on
oop_rasterization: enabled
opengl: enabled_on
rasterization: enabled
skia_renderer: enabled_on
video_decode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
Load (avg) undefined
Memory (System) 15.93GB (6.67GB free)
Process Argv --file-uri=vscode-remote://wsl+Ubuntu-20.04/mnt/d/.de1/Code/workspace.code-workspace --remote=wsl+Ubuntu-20.04 --crash-reporter-id 2459b23f-72eb-4ec8-a6b4-b166f7e6ac7d
Screen Reader no
VM 0%
Item Value
Remote WSL: Ubuntu-20.04
OS Linux x64 5.10.16.3-microsoft-standard-WSL2
CPUs AMD Ryzen 5 2600 Six-Core Processor (12 x 3393)
Memory (System) 12.43GB (10.35GB free)
VM 0%
Item Value
Remote WSL: Ubuntu-20.04
OS Linux x64 5.10.16.3-microsoft-standard-WSL2
CPUs AMD Ryzen 5 2600 Six-Core Processor (12 x 3393)
Memory (System) 12.43GB (10.34GB free)
VM 0%
A/B Experiments 
vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyl392:30425749
pythontb:30283811
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscoreces:30438340
pythondataviewer:30285071
vscod805:30301674
pythonvspyt200:30340761
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
vsc1dst:30438360
pythonvs932:30410667
wslgetstarted:30433507
vsclayoutctrc:30437038
vsrem710:30416614
vscop841cf:30438916
dsvsc009:30440023
vsbas813:30436447
vscscmwlcmc:30438804
helix:30440343

Activity

  1. github-actions
    added
    triage-neededNeeds assignment to the proper sub-team
    on Feb 19, 2022
  2. Karthik Nadig (karthiknadig)
    assigned
    Kim-Adeline Miguel (kimadeline)
    on Feb 22, 2022
  3. Karthik Nadig (karthiknadig)
    added
    area-testing
    bugIssue identified by VS Code Team member as probable bug
    triage
    and removed
    triage-neededNeeds assignment to the proper sub-team
    on Feb 22, 2022

  4. kimadeline commented on Feb 23, 2022

    @kimadeline
    Kim-Adeline Miguel (kimadeline)
    on Feb 23, 2022

    Hi Cnoor0171, thank you for reaching out.

    As you experienced, right now we read test output from stdout/stderr, and this is very brittle. We are updating this mechanism as part of #6594 and #17242.

    Secondly, the throwOnStdErr option is currently set to true when running test discovery, which we shouldn't do.

  5. Kim-Adeline Miguel (kimadeline)
    added
    needs PR
    reason-preexisting
    and removed
    triage
    on Feb 23, 2022
  6. Karthik Nadig (karthiknadig)
    removed
    reason-preexisting
    on May 4, 2022
  7. Kim-Adeline Miguel (kimadeline)
    assigned
    Karthik Nadig (karthiknadig)
    and unassigned
    Kim-Adeline Miguel (kimadeline)
    on Jun 13, 2022
  8. github-actions
    removed
    needs PR
    on Aug 9, 2022
  9. Kartik Raj (karrtikr)
    added
    needs PRReady to be worked on
    on Aug 9, 2022

  10. eleanorjboyd commented on Jul 14, 2023

    @eleanorjboyd
    Eleanor Boyd (eleanorjboyd)
    on Jul 14, 2023

    still exists on rewrite- will need another PR to fix.

  11. Eleanor Boyd (eleanorjboyd)
    assigned
    Eleanor Boyd (eleanorjboyd)
    and unassigned
    Karthik Nadig (karthiknadig)
    on Sep 6, 2023

  12. eleanorjboyd commented on Sep 6, 2023

    @eleanorjboyd
    Eleanor Boyd (eleanorjboyd)
    on Sep 6, 2023

    tried this on the rewrite on insiders and it no longer shows the test as failed on discovery. I was also able to run the test as follows and it works passes.

    def test_passing():
    sys.stderr.write("something something something")
    assert 1 == 1

    please comment if I tested it incorrectly or if you are still seeing the issue.

  13. Eleanor Boyd (eleanorjboyd)
    closed this as completedon Sep 6, 2023
  14. github-actions
    removed
    needs PRReady to be worked on
    on Sep 6, 2023
  15. github-actions
    locked as resolved and limited conversation to collaborators on Oct 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area-testingbugIssue identified by VS Code Team member as probable bug

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions