[AlexWaygood]

This PR:

• Moves the logic for running mypy on the test cases from tests/mypy_test.py to a separate script, tests/regr_test.py.
• Adds the necessary logic in order to be able to have test cases for third-party stubs.
• Moves logic common to tests/mypy_test.py and tests/regr_test.py into tests/colors.py, and renames tests/colors.py to tests/utils.py.
• Adds a new check to tests/check_consistent.py, to enforce the use of # pyright: reportUnnecessaryTypeIgnoreComment=true comments in third-party test cases. These are essential if we want to have our tests against false-negatives work with pyright.
• Updates the relevant documentation to account for the new test file.
• Adds a new job to the tests.yml GitHub workflow, to run the new test in CI.
• Adds a simple proof-of-concept test case for requests, as a regression test for requests: allow passing a list of tuples for files #7998.
• Closes Figure out a way to have test cases for third-party stubs #8569

[srittau]

Not reviewed yet, but the list above sounds great!

[AlexWaygood]

Okay, this should now be ready for review! Getting this to work both locally and in CI was a bit of a wild ride.

The second commit deliberately introduces errors in stdlib test cases and third-party test cases, to show what it looks like in CI when mypy and pyright fail on those test cases. Unfortunately, it looks like if there are failures in the stdlib test cases, pyright won't then go on to check the third-party test cases. I'm not sure that's fixable without rethinking how we run pyright in CI:

• pyright run on commit 2: https://github.com/python/typeshed/runs/8234439208?check_suite_focus=true
• mypy run on commit 2: https://github.com/python/typeshed/runs/8234442216?check_suite_focus=true

The third commit fixes the errors in the stdlib test cases, so we can see what it looks like when there are only errors in the third-party test cases:

• pyright run on commit 3: https://github.com/python/typeshed/runs/8234492908?check_suite_focus=true
• mypy run on commit 3: https://github.com/python/typeshed/runs/8234496168?check_suite_focus=true

The fourth commit fixes the errors in the third-party test cases, so we're back to how it was in commit 1.

[JelleZijlstra]

Thanks! A few nits.

[srittau]

LGTM, although I skipped some of the code for time constraints. One optional remark and some comment fluff.

[srittau]

Maybe we should move this into utils.py as well? Or maybe add a config.py?

[AlexWaygood]

I think it might be better to keep this per-test. We already have one test that supports a narrower version range than this (with good reason, since we're using syntax new to 3.9 in many of our tests).

typeshed/tests/typecheck_typeshed.py

Line 16 in e4d0d3d

SUPPORTED_VERSIONS = ("3.11", "3.10", "3.9")

And we started testing 3.11 with mypy_test.py much earlier than we started testing it with stubtest_stdlib.py.

[github-actions]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

[AlexWaygood]

Thanks for the reviews! I'm going to land this now, but happy to address any further feedback anybody has in follow-up PRs :)