Repository navigation
Slow typechecking on nested TypedDict with union members #17231
Description
Bug Report
For Pulumi we are looking into generating types using TypedDict to model cloud APIs. For example for Kubernetes we have something representing a Deployment.
class DeploymentArgsDict(TypedDict):
api_version: NotRequired[Input[str]]
kind: NotRequired[Input[str]]
metadata: NotRequired[Input['ObjectMetaArgsDict']]
...Pulumi has a notion of inputs and outputs, and the Input type used in the above example looks like this:
Input = Union[T, Awaitable[T], Output[T]]
class Output(Generic[T]):
passOutput does a lot things, but for the purposes of this repro all that matters is that its a generic type.
The K8S types can nest pretty deeply, and I suspect a combination of having nested literals along with the Union via the Input type is causing slowness here.
Example:
d: DeploymentArgsDict = {
"metadata": {
"name": "nginx",
},
"spec": {
"selector":{
"match_labels": {}
},
"replicas": 1,
"template": {
"metadata": {
"labels": {}
},
"spec": {
"containers": [{
"name": "nginx",
"image": "nginx"
}]
}
}
}
}
If I drop Awaitable[T] from the union to reduce it to two members, typechecking completes in 2 seconds. With it present, it takes 40 seconds.
This is a simplified example, and the actual code has another union layered on top. In that case we run out of memory.
To Reproduce
I have created a repro here https://github.com/julienp/typeddict-performance
Expected Behavior
It takes a second or two to typecheck.
Actual Behavior
It takes ~40 seconds on my machine
Your Environment
- Mypy version used: 1.10
- Mypy command-line flags: none
- Mypy configuration options from
mypy.ini(and other config files): none - Python version used: 3.12.2
Activity
julienp commentedon May 13, 2024
Ran some more tests with a larger set of types, and it looks like the issue might be memory related. I am seeing python max on memory on my system, causing heavy swapping, while the process sits at 100% CPU, probably GCing constantly.
justinvp commentedon May 30, 2024
Any idea what the root cause could be or how we could workaround it, or even help contribute a fix?
We'd like to improve Pulumi's Python SDKs by supporting TypedDict, but this performance issue means we'd have to workaround it for Mypy users, likely by conditionally typing these as untyped dictionaries for Mypy, which is rather unfortunate.
east825 commentedon Sep 25, 2024
Because of the number of
Inputunions used in type annotations there and a huge number of TypedDict involved, some resulting fully expanded TypedDicts are humongous. For instance,DeploymentArgsDictdepends on declarations of all the following TypedDicts and the resulting complete type contains 27M+ (!) internal types. It has no self-references and recursively defined TypeDicts, though, as far as I can tell. PyCharm inference also suffers from this. I'm wondering how Pyright approaches such TypedDict trees.erictraut commentedon Sep 25, 2024
Does mypy internally expand all of these TypedDict definitions? If so, I'm curious why. Pyright internally builds one object for each class. There are only 438 of them in the code sample, which isn't that many. Each internal object refers to the other objects as needed. It doesn't do any expansion.
east825 commentedon Sep 26, 2024
To be clear here, I didn't check how Mypy internally represents such types. In PyCharm, we represent TypedDicts as
dict[str, UnionOfTypesOfAllFields]for some type checks, and constructing this union of all field types (recursively), simultaneously expanding type aliases, leads to such combinatoric explosion. But since there are memory problems, I guess the root cause might be somewhat similar.JukkaL commentedon Sep 27, 2024
There seem to be some easy improvements we can make to speed up the handling of nested TypedDicts. I don't think there's any deep reason why they'd have to be this slow. I'll look into this -- if it's easy enough, the next mypy release (to be out in a week or two) could include some optimizations.
JukkaL commentedon Sep 27, 2024
#17842 fixes some bottlenecks.
Avoid type size explosion when expanding types (#17842)
julienp commentedon Feb 9, 2026
I finally got around to testing this again, and it looks like this is no longer an issue for generated pulumi SDKs like
pulumi_aws.The repro linked in the issue above now typechecks quickly with mypy 1.12 and above.
Remove workaround for slow typechecking with MyPy and PyCharm
Remove workaround for slow typechecking with MyPy and PyCharm
Remove workaround for slow typechecking with MyPy and PyCharm (#21722)