Repository navigation
What to do about inferring types from multiple scopes? #4492
Description
Typically mypy infers the type of the variable from the first assignment. In some cases mypy infers the type from two assignments. The first assignment creates internally a partial type, which is missing some information. The second assignment completes the type. Example:
x = [] # x gets a partial List type
x.append(1) # x gets complete type List[int]In some cases the assignments can be in two different scopes. Example:
x = []
def f() -> None:
x.append(1)
# type of x is List[int]This feature has apparently been supported a while, but it has some issues:
- This behavior is somewhat messy to support in fine-grained incremental mode (Refresh of partial type produces the wrong type in fine-grained incremental mode #4464).
- It can make it hard to figure out the inferred type of a variable, since the two assignments can be separated from each other by hundreds of lines and many functions.
- Moving functions around can affect the type of a module/class variable in cases where there are two potential functions that can complete a partial type.
- If the second assignment is missing or removed, and the first assignment initializes the variable with
None, the inferred type will becomeNone, which is usually not what the user wanted. - It seems a bit fragile with some issues that took a long time to notice (Inconsistent types for variable with partial type #4484, Inference from unannotated functions #4455, Invalid type inferred for attribute initialized to
Nonein class body #4416).
There are a few alternatives which might be reasonable, instead of the current behavior:
- Require an annotation for the variable in cases such as the above, where the assignment that completes a partial type is in a different scope. This would break compatibility with previous mypy releases. Based on a quick experiment, the S internal codebase at Dropbox would require about 40 additional type annotations if we make this change.
- Ignore assignments in nested scopes and assume
Anyfor any missing partial type information. For example, if the initial assignment isx = []and the only additional assignments are in nested scopes, inferList[Any]as the type ofx. This way the type ofxis defined entirely by the contents of a single scope, which solves the above problems. This also introduces an additional problem -- implicitAnytypes. We might want to have a strictness flag that falls back to requiring a type annotation (as in option 1 above).
Currently my biggest concern is fine-grained incremental mode. There it may be possible to support the current behavior by introducing a new concept: "bound targets". I'll add a writeup about this to #4464.
@gvanrossum @msullivan @ilevkivskyi Thoughts?
Activity
ilevkivskyi commentedon Jan 19, 2018
I am in favor of option 1. I think I mentioned this some time ago, and I didn't change my mind. My argument is simple: I don't like implicit
Anytypes. Currently PEP 484 defines small set of situations whereAnyis assumed (like non-annotated functions and missing generic arguments) and we have flags to prohibit those. Adding one more source of implicitAnys seems not worth it (especially that users might not be aware of it and will therefore get false negatives).Also the number 40 seems relatively small.
ilevkivskyi commentedon Jan 19, 2018
An additional argument for option 1 is that the function where the second assignment happens may be non-annotated. In this case the inferred type will be e.g.
Optional[Any]. This can lead to some hard to spot false negatives. This has been also discussed in #4455gvanrossum commentedon Jan 19, 2018
JukkaL commentedon Jan 25, 2018
Ok, let's go with option 1.
JukkaL commentedon Jan 25, 2018
Turns out that my figure of about 40 annotations was too optimistic -- I wasn't counting all kinds of partial types. A better figure is about 80 annotations for the S codebase. I think that the adaptation work is still reasonable and option 1 is still the best one.
JukkaL commentedon Jan 26, 2018
I found some further cases. It looks like the C codebase needs around 40 annotations and S around 140. The most common case in S is an attribute initialized to
Nonein the class body.I'm about half way through the migration of Dropbox codebases. It has been enough work that it might make sense to make it easier. First, we could have a per-file option that ignores missing annotations (and will just infer
Anycomponents for missing collection item types, and so on). Second, we could try to infer a type and suggest it in some of the most common cases. Example:gvanrossum commentedon Jan 26, 2018
Off-line we decided on a compromise solution. IIRC it's something like add a global flag to enable or disable the current behavior, and force disabling the behavior in fine-grained mode; and when the current behavior is disabled, show an error with a suggestion for the type. E.g.
This would either be allowed (default in non-fine-grained mode) or produce an error on
x = Nonewith the suggested typeint. The suggested types would be formatted the same as types in messages fromreveal_type(), and the phrasing would make it clear that the suggestion is just that, not a guarantee.Fine-grained: Don't infer partial types from multiple targets (#4553)
JukkaL commentedon Mar 8, 2018
Closing in favor of #4703 which is about suggesting type annotations. Requiring an explicit annotation was implemented in #4553.
Fine-grained: Don't infer partial types from multiple targets (python…