Repository navigation
Type erasure and related issues #79
Description
Activity
We (me, Mark Shannon and Andrey Vlasovskikh) discussed this yesterday and the consensus was that
Node[t]()should be disallowed in expressions (this also applies to things likeNode[t].foo(), though we weren't sure whether we can reasonably modify the implementation to not allow that.Given the expense of creating a new class, it might be worth caching these things in the
typingmodule. That way we don't need to explicitly disallowNode[T]()ast it will only create one new class object perT, not per evaluation.I like that idea. Maybe I won't get to implementing it before beta 1 though.
I have another issue with supporting
Node[T](). What ifTis a type variable? At runtime, the value of a type variable is unknown. In a function annotation, a type variable implies a generic function, and the type variable applies to the function as a whole, so there is no such issue and introspection works consistently. At runtime, the type variable inNode[T]()applies to a single function invocation whereThas some value (though only statically), so conceptually it should be bound to this particular value, but due to type erasure, the value is not available at runtime.Example:
... class Node(Generic[T]): def __init__(self, x: T) -> None: self.value = x T = TypeVar('T') def f(x: T) -> Node[T]: return Node[T](x) n1 = f(1) # Runtime type Node[T], but should arguably be Node[int] n2 = f('x') # Also Node[T], but should be Node[str]My argument against storing the type argument in instances is primarily based on two issues:
- Type variables cannot be used usefully (their value is unknown), so the information about type arguments is only partial and thus can't be usefully relied on. Instances created within generic functions may have different properties from instances created in non-generic functions, which non-uniform and ugly.
- If an instance is created using
Node(...), the type argument value is missing. Again, this means that runtime introspection of type argument values is not very useful since the value is not always there, and the behavior is inconsistent.
These are more minor issues:
Node[T](), even with caching, seems like a somewhat expensive operation (we could easily benchmark how fast it is compared to justNode(), though).List[int]()would not be supported (and generally any std library class would not support indexing, unless it is defined intyping), which makes these types inconsistent from built-in types.type(node) is Nodedoesn't work forNode[int]()instances, which means that code that used to work when it was dynamically typed may break once it is adapted to static typing (which may introduceNode[t]()etc.).
OK, unless there are further objections, let's disallow instantiating parametrized types.
How about we make
Node[t]()return an instance whose__class__is justNode(notNode[t])? Just so if there are no other ways the type checker can guess the type (e.g. it's an empty container) we don't have to resort to a# type: Node[t]comment? This could work for concrete types as well as type variables.We could even go back and make
List[t]()return an empty builtinlistinstance?That could be nice -- that's how mypy used to work, and it was somewhat helpful. These are the tradeoffs as I see them:
List[t]()(etc.) is shorter to write than a type comment and arguably looks less noisy.List[t]()is less efficient than[]. We should measure by how much and if it's a lot slower we may want to reconsider.- There's now more than one obvious way to do things. Some people may prefer a type comment while others may prefer
List[t](). Should one of these be the recommended way in most cases? For example, we should recommend using a type comment for initialization andList[t]()in subexpressions where a type comment isn't valid. Variable/attribute initialization is clearly the most common use case for these.
I wonder if the real utility of
List[t]()(etc.) is when theList[t]part is actually a type alias. I expect that reading e.g.List[Employee]()will feel pretty cryptic (there's so much punctuation that less-experienced readers are likely to either skip it or be stopped in their tracks trying to understand it), but e.g.EmployeeList()feels more readable.This needs text in the PEP describing what we decided to do. I think it's this (which is already implemented in typing.py):
Node[t]()(for some concrete type t) as well asNode[T]()andNode()are all allowed, but don't record the type parameter (if any) at runtime.- Type checkers should capture the type parameter and proceed to check uses leading from this instantiation accordingly.
FWIW I also need to fix this in typing.py -- currently
Node().__class__isNode[T],Node[int]().__class__isNode[int], etc. (I.e. I was wrong in the previous comment that this is already implemented correctly.)- added a commit that references this issue
on May 20, 2015 I found the section in the PEP (added in c29a9b9) unclear to read as someone not very familiar with type systems (coming from C and Python).
Two things are confusing to me:
- What part is talking about the statically inferred type, and what part is talking about the runtime type?
- What's the purpose of instantiating such a generic?
After reading it a few times, here is what I think it means:
- the type of x is
Node[Any]. refers to the statically inferred type, as it otherwise conflicts with At runtime the type is not preserved, and the observable type of x is
justNode. - Instantiating it will just create a
Nodeobject (as if it didn't inherit fromGeneric) and the extra infos are only for the typechecker.
Is this correct? Maybe the text could be clarified a bit, though I'm not sure how exactly.
1 remaining item
Done in rev b5e4e3d.
I'm reopening this. The PEP (https://www.python.org/dev/peps/pep-0484/#instantiating-generic-classes-and-type-erasure) explicitly allows
Node[int](), but mypy explicitly disallows it, with this error:error: Generic type not valid as an expression any more (use '# type:' comment instead)The discussion in this issue was somewhat hurried and went back and forth a few times. What should we do? Fix mypy or change the PEP?
I do not see why
Node[int]()should be prohibited. Probably, the only downside is speed, but people who want speed simply will not use this particular possibility. On the other hand assume that I haveclass Box(Generic[T]): def __init__(self, value: T) -> None: self.value = value
If I want to be able to find type errors at instantiation, then I have two options:
x = Box[int]('a string') # OOPS!
or alternatively
x = Box('a string') # type: Box[int]
The second option looks a bit ugly to me. It would be even more ugly if I want to type annotate something like
if Box('a string') in box_list: # type: Box[int] ...
Although there might be no serious reason to recommend the first option, I do not see any reason to prohibit it.
The main reason to prohibit
Node[int]()is indeed that it's expensive. In fact, it's really expensive. Like 100x the cost ofNode()(if Node is generic; otherwise it's more like 250x).Why is it so slow? Because each time you call
Node[int]it creates a new class object. And class objects just aren't meant to be created that frequently.However you can still create a type alias for
Node[int]and call it. E.g. the following works in mypy (and is barely slower than using a type comment):IntBox = Box[int] x = IntBox(0) # okay y = IntBox('') # errorType alias is a nice compromise here. I would even say that using type aliases could improve readability in this case. Finally, the facts that this already works in mypy and it is fast mean that this is a good option that we could recommend.
- Maybe you could update mypy's error message to hint at that?
Explicitly making a type alias such as
Node[int]be a valid thing to call seems reasonable, but I think that we should erase type variables -- my preference would be to makeIntBox(0)construct the same object asBox(0)at runtime in the above example. Allowing something likeBox[int]to be called within an expression seems questionable unless we can make it at least reasonably fast.- The alias solution already works -- mypy understands it and the runtime implementation does erase the type variable. The question is purely about whether the PEP should disallow `Box[int]()` or whether mypy should start allowing it (at runtime it also works, but is really slow). I think that the best solution really is to disallow `Box[int]()` in the PEP (but continue to support it via type aliases) since the `Box[int]` part of the expression is just always going to be expensive (even if we could speed it up a bunch with caching). If you agree, I'll update the PEP and maybe you can update mypy to also suggest an alias in the error message.
Sorry, I was quite busy today, @gvanrossum I agree with everything what you said, I propose to change the error message to the following one "Generic type is prohibited as a runtime expression (use a type alias or '# type:' comment)". I have made a PR at mypy python/mypy#1326
This entire section in the PEP is questionable (showing how little I understood when I wrote it):
Now there are two ways we can instantiate this class; the type inferred by a type checker may be different depending on the form we use. The first way is to give the value of the type parameter explicitly -- this overrides whatever type inference the type checker would otherwise perform:: x = Node[T]() # The type inferred for x is Node[T]. y = Node[int]() # The type inferred for y is Node[int]. If no explicit types are given, the type checker is given some freedom. Consider this code:: x = Node() The inferred type could be ``Node[Any]``, as there isn't enough context to infer a more precise type. Alternatively, a type checker may reject the line and require an explicit annotation, like this:: x = Node() # type: Node[int] # Inferred type is Node[int]. A type checker with more powerful type inference could look at how ``x`` is used elsewhere in the file and try to infer a more precise type such as ``Node[int]`` even without an explicit type annotation. However, it is probably impossible to make such type inference work well in all cases, since Python programs can be very dynamic. This PEP doesn't specify the details of how type inference should work. We allow different tools to experiment with various approaches. We may give more explicit rules in future revisions.I would like to rewrite all of this it to clarify:
x = Node()infers the type of x asNode[Any]- if the typevar occurs in the signature of
__init__we infer the type from that instead x = Node() # type: Node[X]infers the type of x asNode[X](for some concrete type X)x = Node[X]()is invalidA = Node[X]followed byx = A()is valid and infers the type of x asNode[X]
(Also added to the peps repo, as rev 1ad9e828ca36.)
We said it shouldn't be possible to instantiate
ListorList[t](i.e.,List()orList[t]()), but the implementation allows this (wheretcan be a type variable or a concrete class). This should be disallowed (and made sure it's documented in the PEP).However, for user-defined generic classes it should definitely be possible to write node
Node()and probably alsoNode[t](). We must decide whether the latter is allowed, and if so whether the object it returns hasNodeorNode[t]as its__class__.Also note that if the
Nodeclass is declared generic in a stub file, all uses ofNode[t]must appear in string quotes (i.e. as forward references) since in this case the runtimeNodeclass is not parametrized. (See #60.)Mark Shannon has proposed an intermediate option, using a class decorator like
@implements(Generic[T]), which would make the fact thatNodeis parametrized knowable at runtime but without definingisinstance(),issubclass()orNode[t]. This would be a third way to do it, not replacing stubs or inheriting fromGeneric[T]. It could possibly also save space (since in CPython class objects such as created byNode[t]use a lot of memory).