Auto-discovery of .test files

Currently all the .test files (used for mypy self tests) are enumerated in the test case runners, such as in mypy/test/testcheck.py. This is error-prone and can be confusing.

It would be better to do auto-discovery of test files using file name prefixes. For example, all the files with names like check-*.test would be executed by mypy.test.testcheck.

(Additionally, if there is an unrecognized .test file, it would be nice to generate an error and give hints about how to fix the issue. Addressing this is optional for now, however, and may be a bit tricky.)

@ethanhs

I am interested in looking into this, as I was a bit surprised that the testpep561.py file parsed the pep561.test file by direct reference (as do other tests I presume?). As I've read they do this by specifying the files property in a DataSuite-inheriting class. I've worked with code running in this extract (testpep561.py):

class PEP561Suite(DataSuite):
    files = ['pep561.test']

    def run_case(self, test_case: DataDrivenTestCase) -> None:
        test_pep561(test_case)

One would expect one (or a couple extended) classes(s) to parse all the tests and generate a recognizable pytest output.

It's going to require a bit more understanding on my side, although I gained some experience through #8524 and @ethanhs his work in #8571. So just for clarity's sake:

We are interested in intercepting pytest autodiscovery for data-driven tests (any class extending Suite/DataSuite/etc.) . Any pointers to which class to start studying if so? (File: data.py or helpers.py possibly).
We are interested in combining unit tests starting with foo by regex'ing foo-*.test files.
We need some scanning or mapping state of which tests are picked up only once by a module (f.e. mypy.test.testcheck as your example goes). I'd expect this to be more strict: we must enforce a .test file to map to one test*.py file only. Only then can we make sure the data-driven tests are run by at most one module. A second requirement of course is that the unit test maps to the right test module. A third is that if the test-data can regex to multiple modules, an error must be thrown. Does this sound complete/correct?
We'd like some way of generating a mapping of the data unit tests in a custom discovery class, with 'missed_cases' and 'multiple_mappings' as output, but it is expected to by tricky because it can generate wrong information/advice or has some edge cases.

@JukkaL

@JukkaL it'd be awesome to hear your thoughts so I can get to work.

Just a 'go-ahead' is fine for now.

@ethanhs

I will work on this, although any comment would be great. @ethanhs do you have any thoughts? It isn't a small endeavor this time, but pretty similar to data-driven tests like we worked on earlier

@JukkaL

I'm afraid I'm not sure the particulars of what @JukkaL has in mind, so it'd be best to ask him.

@JukkaL

@JukkaL I'd gladly hear about whether this issue can be acted upon

I'd like to work on this issue.
From what I understand, we need the list of .test files to be auto generated. I am thinking of using something like pathlib.Path().rglob("prefix*.test") to do so. What do you think?

@ethanhs

Up for grabs, @ethanhs @JukkaL could you help this fella out?

Hello Sir ,

I was trying to fix auto-discovery of .test files #8650

Here is the script

# List of files that contain test case descriptions.

typecheck_files = [f for f in os.listdir("../../test-data/unit") if re.match(r'check-\w+-?\w+?.test',f)]

# Tests that use Python 3.8-only AST features (like expression-scoped ignores):
if sys.version_info < (3, 8):
    typecheck_files.remove('check-python38.test')
if sys.version_info < (3, 9):
    typecheck_files.remove('check-python39.test')

# Special tests for platforms with case-insensitive filesystems.
if sys.platform not in ('darwin', 'win32'):
    typecheck_files.remove('check-modules-case.test')

But there was path not found error. Can you exactly tell me from where would it be called ?
As seen in ReadMe , you have called many functions as

$ pytest -n0 mypy/test/testcheck.py

So is that the default location from where we can call it ? If yes ,then maybe this would work :

# List of files that contain test case descriptions.

typecheck_files = [f for f in os.listdir("test-data/unit") if re.match(r'check-\w+-?\w+?.test', f)]


# Tests that use Python 3.8-only AST features (like expression-scoped ignores):
if sys.version_info < (3, 8):
    typecheck_files.remove('check-python38.test')
if sys.version_info < (3, 9):
    typecheck_files.remove('check-python39.test')

# Special tests for platforms with case-insensitive filesystems.
if sys.platform not in ('darwin', 'win32'):
    typecheck_files.remove('check-modules-case.test')

Please Help !!

@JukkaL

Hello, I'd like to solve this issue, although many have already solved this prior to me.
@JukkaL @davidzwa Can anyone reach out, I have a few pointers. Also, it'd be great to start my contribution journey from this issue of mypy.

Thanks!

testcheck: auto-discovery of .test files

Tests autodiscovery replacing hardcoded file list (#13143)

Navigation Menu

Repository navigation

Auto-discovery of .test files #8650

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Auto-discovery of .test files #8650

Description

Activity

davidzwa commented on Apr 10, 2020

davidzwa commented on Apr 15, 2020

davidzwa commented on Apr 28, 2020

emmatyping commented on Apr 29, 2020

davidzwa commented on May 6, 2020

iamsdas commented on Dec 20, 2020

davidzwa commented on Dec 20, 2020

MiHarsh commented on Feb 8, 2021

SwagatSBhuyan commented on Nov 3, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions