Test discovery should be fault tolerant and display all successfully imported files

Currently, pytest discovery fail for the entire project if a single import is missing on a single file:

E   ModuleNotFoundError: No module named 'xyz'
ERROR my_test.py
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
Exception: pytest discovery failed (exit code 2)

Only 1/600 file failed, but VS Code does not show the 599 tests which were correctly discovered.

Many other project have "optional" deps which add extra features. Those extra deps shouldn't be required when the full project do not need to be tested.
My project has many independent modules, each requiring a specific set of dependencies (pip install .[module0]). Contributors should not be forced to install the full dependencies of all modules / features when they only want to test a subpart.

Related issues:

Those issues are not duplicate but seems related:

Perform test discovery per-file when a file is updated/added. #4586: Perform test discovery per-file when a file is updated/added: This issue focus about performances, as re-discovering the full set of files each time is slow.
Test discovery fails if package is not installed #10469: Test discovery fails if package is not installed: This is about a specifc case where a specific import was not correctly detected.

Were you able to get pytest or unittest to show tests via command line, even with import errors?

Yes, pytest --collect-only correctly detect the valid tests:

============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/epot/__A__/Code/datasets, configfile: pytest.ini
plugins: xdist-1.34.0, forked-1.3.0
collected 1685 items / 1 error / 1684 selected

<Package tensorflow_datasets>
  <Module import_test.py>
    <UnitTestCase ImportTest>
      <TestCaseFunction test_import>
      <TestCaseFunction test_session>
  <Module import_testing_test.py>
    <UnitTestCase ImportTestingTest>

...  # The 1684 detected tests are correctly displayed

=========================== short test summary info ============================
ERROR tensorflow_datasets/scripts/documentation/build_api_docs_test.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
======================== 5 warnings, 1 error in 35.20s =========================

Additionally, pytest --continue-on-collection-errors correctly execute the valid tests.

Shouldn't the info needed label be removed ?

--continue-on-collection-errors makes test discovery bit better when there are issues with loading test files. However, adding support that will require a bit of work in the current test adapter. We have plans to update the adapter to be more robust. We also have to see if similar support can be provided on other supported test frameworks.

I am facing the same error with the plugin. For manual test discovery in a terminal I can fix the error by executing:

pip install -e LOCAL_PACKAGE_PATH
source ~/ROS2_WORKSPACE

Can I prepend these command somehow to the plugin, so that I could use the test discovery?

Hello! This should be fixed in the rewrite which is currently underway! The meta-issue is here and that will track progress but I will keep this issue open as well until resolved. Thanks!

display errors and tests on discovery (#21629)

Hello! A fix for this is out on insiders! If someone could tests and let me know it worked that would be great!

Verification steps:

take following code and make two files with each block
run pytest or unittest discovery
check that there is a node with the tests that can be discovered and one saying there was a failure

def test_example():
    assert 1 == 1

def test_two():
    assert True

import unittest
import foo #doesn't exist so will error

class OtherTest(unittest.TestCase):
    def testtwo(self):
        self.assertEquals(add(4, 5), 9)

if __name__ == "__main__":
    unittest.main()

Merged PR posit-dev/positron-python#151: Merge upstream as of 2023 Ju…

Navigation Menu

Repository navigation

Test discovery should be fault tolerant and display all successfully imported files #13301

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Test discovery should be fault tolerant and display all successfully imported files #13301

Description

Activity

karthiknadig commented on Aug 6, 2020

Conchylicultor commented on Aug 6, 2020

Conchylicultor commented on Aug 10, 2020

karthiknadig commented on Aug 10, 2020

3 remaining items

fnobis commented on Jan 3, 2023

eleanorjboyd commented on Jan 6, 2023

eleanorjboyd commented on Jul 21, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions