Repository navigation
Figure out a way to have test cases for third-party stubs #8569
Description
We currently only have test cases for our stdlib stubs, and don't currently have the infrastructure set up to allow us to add test cases for any of our third-party stubs. This has come up a few times now, so it would be good to figure out a way to get this to work.
(This issue relates to the test cases in our test_cases directory. Currently we test these samples with mypy and pyright.)
Activity
AlexWaygood commentedon Aug 19, 2022
As @srittau wrote in #8505 (comment):
kkirsche commentedon Aug 19, 2022
To get us started on this, I think it would make sense to outline a few stubs that would be good candidates for this so that we can leverage those stubs when prototyping or suggesting approaches.
sobolevn commentedon Aug 29, 2022
In
django-stubswe run type checking on django tests.It is not ideal at all, but our project is quite old at this point, so we didn't have much options back in the days.
Link: https://github.com/typeddjango/django-stubs/blob/master/scripts/typecheck_tests.py
AlexWaygood commentedon Aug 29, 2022
Some updates on my plan for this:
tests/mypy_test.py. I'd like to get the refactoring oftests/mypy_test.pyrelated to mypy_test: Areexcludeandfilteruseful? #8139 done first, before I tackle this. That should be done soon, then I'm planning to move onto this.requestsmight be a good candidate, given the number of issues we had filed earlier in the year after seemingly innocuous changes caused serious regressions for users:files#7998Flask-SQLAlchemymight also be a good candidate for some third-party test cases: theModelclass is pretty unusual, in that at first glance it seems like a good candidate to be made generic, but if you make it generic, that prohibits use of thequery_classattribute as a class variable. See:Flask-SQLAlchemy: Make model query non-generic #8455Akuli commentedon Aug 29, 2022
Most of the requests problems were introduced when mypy_primer was broken for third-party stubs.
AlexWaygood commentedon Aug 29, 2022
Yeah, mypy_primer is and always will be our best tool for catching regressions. The
test_casesdirectory will always be a secondary tool.kkirsche commentedon Aug 31, 2022
I can help write tests for some of aspects of Flask-SQLalchemy and SQLAlchemy, depending on the coverage goals. To the question of what it is that we're testing, what should these tests actually be covering?
What I mean by that is meant to be the specifics not the ideological discussion covered in #8505, e.g. are we testing that the behavior is functional (so using an in-memory or external database service to verify both the actions and the types) and the types pass, or only that the types pass for idiomatic use cases (essentially acting as a form of documentation while providing a bit more security in updates)?
If using external services, such as the database in the case above or an external service's URL for requests, what external resources does the team want to use for these, as this has implications on test stability if there are downstream outages or issues?
AlexWaygood commentedon Sep 1, 2022
Thanks for the offer! I appreciate it.
As stated in the
test_casesREADME, we're not really going for 100% code coverage at all with these test cases; it's just really meant to be for exceptional cases where the stubs are quite tricky to get right, and/or where we've run into specific difficulties in the past. But if you have specific proposals for certain areas of SQLAlchemy or Flask-SQLAlchemy that you think could do with some test cases, we can discuss it once we've got the infrastructure set up.The test cases being discussed in this issue shouldn't need access to any external services. The code in these test cases is never actually run; we just run the type checkers on the code so that we can assert that type-checking errors are emitted in the correct places. This isn't a comprehensive way of testing stubs, but it's not meant to be; if we want to discuss more comprehensive ways of testing stubs, we should do that elsewhere :)
kkirsche commentedon Sep 2, 2022
OK, cool.
So in terms of infrastructure, what you mean by that is:
The above assumes that we'd use
stubs/*/@testsfor the files to be tested. Potentially as a sub-directory to avoid mixing test files with configuration files, as would occur due to things likestubtest_allowlist.txtfiles and whatnot. If agreed to in the scoping, potentially separate into@tests/configand@tests/usage.Is there anything I'm missing / am I over simplifying what's required?
AlexWaygood commentedon Sep 2, 2022
That pretty much covers it, yeah. I've assigned this issue to me to indicate that I'd like to work on these steps :)
AlexWaygood commentedon Sep 2, 2022
One complicating factor is:
reportUnnecessaryTypeIgnoreCommentsetting for the wholetest_casesdirectory #8506If we can figure out the right way to organise our
pyrightconfig.jsonfiles, it should really be pretty easy to have pyright'sreportUnnecessaryTypeIgnoreCommentsetting enabled by default for the wholetest_casesdirectory. If we want to have third-party stubs instubs/*/@tests/test_casesdirectories, however, rather than in the top-leveltest_casesdirectory, then having that pyright setting enabled by default for all our third-party test cases might be a little tricky.AlexWaygood commentedon Sep 7, 2022
Okay, here's a PR! Reviews welcome: