Repository navigation
Protocols (a.k.a. structural subtyping) #11
Description
Activity
It's meant to be used for structural subtyping. It is quite similar to ABCs, but neither is a replacement for the other. The details of how to use
Protocolare still poorly defined, as the mypy type system does not know about structural subtyping yet. However, adding support for structural subtyping shouldn't be too difficult, once we figure out the semantics.Here is an example of how
Protocolcould be used:class SupportsFileno(Protocol): @abstractmethod def fileno(self) -> int: pass def f(file: SupportsFileno) -> None: id = file.fileno() # Okay ... f(open('foo')) # Okay, since file objects have fileno()!There is no need to explicitly declare that file objects implement
SupportsFileno(the type checker would infer this automatically). This is unlike ABCs, where base classes have to be explicitly defined/registered.The current implementation of
Protocolis just a proof of concept, as is the implementation ofoverload. A production implementation would probably have to be optimized, and there are probably corner cases that aren't handled yet.Protocols look to me like a pretty reasonable way of implementing structural typing, which is definitely something that we're hoping to see in the PEP. This design is necessarily verbose; an in-line, anonymous definition using dictionaries, like
def f(file: {'fileno': Callable[[], int]}) -> None: ...is a bit more concise when the protocol/structural type is small. This would conflict with the current proposal to use dictionaries in annotations to represent current non-type uses of annotations (https://github.com/ambv/typehinting/blob/master/pep-NNNN.txt#L194), however see #26 for thoughts on alternatives.
I haven't seen any evidence for protocol types being used/useful all over the place, so my current working hypothesis is that having a heavy-weight syntax for protocols only adds a trivial amount of syntactic overhead for the vast majority of programs.
However, the absence of evidence doesn't prove anything -- if somebody finds a few counterexamples I'm happy to change my mind. :)
Maybe we could have a look at some Go code and see how many small, throwaway structural types are used there? Of course, having convenient syntax for a feature may nudge programmers into using the feature more often.
ABCs have runtime instance checks so it's hard for the type checker to use them for structural sub-typing. If it could, that would be the simplest and most elegant solution. Otherwise, could we leave this out for now and look how we could make it work with existing protocol solutions like Zope interfaces?
Jukka, your typing.py has a Protocol class that is more complex than most other infrastructure, and it is used for those generic ABCs (e.g. Sized, Hashable) whose collections.abc counterpart does a runtime structural check. Does this mean that you have now implemented this in mypy? I kind of like the resulting syntax for defining a generic ABC that is supposed to use structural testing:
class Sized(Protocol): @abstractmethod def __len__(self) -> int: pass class Container(Protocol[T]): @abstractmethod def __contains__(self, x) -> bool: pass(Note that Sized is not generic, but Container is.)
Łukasz: I have no experience with zope interfaces. But perhaps they are easy enough to parse for a hypothetical static type checker, and they feel close enough to types/classes that we should try to support them? Can you sketch a simple example?
Quoting Łukasz in python/mypy#539 (comment):
"""
As for explicit protocols, there are some competing (as in: not easily composable) standards here:- ABCs with @AbstractMethod
- Zope interfaces
- https://pypi.python.org/pypi/characteristic/
etc. etc.
As ABCs are built-in, it seems natural to suggest that they should be used for defining interfaces. However, I understand that static analysis (so, the type checker) might not be able to process every abstract class in general.
"""Guido, the machinery in mypy's typing.py has been there for a long time, but it was never fully implemented, and in particular, the type checker doesn't know anything about this. It shouldn't really be there -- everything should just use ABCs, until there is type system support for protocols.
I added a new task for removing
Protocol:
python/mypy#552If protocols (or something similar) will be included in the PEP, I'll update the above issue.
So we have multiple competing ways to spell protocols, but no way to type-check them statically. This feels like something we'll have to leave out of the PEP and come back to later in a separate PEP.
OK, let's remove Protocol from typing.py for now. We should come back to it in a separate PEP, especially that PEP 245 has been rejected and there are ABCs now.
As for Zope interfaces, an interface definition is easy to parse, except for invariants (runnable checks, similar to instancechecks in ABCs). We have to bear in mind that existing interface definitions may sometimes be dynamic, it's Python. That's fine, I think. What the type checker doesn't know, it assumes it's correct.
I see two issues with Zope interfaces:
- arbitrary classes can be externally registered as implementing interfaces, just like ABCs can have external classes registered to them (so we'd have the same problems with this)
- adaptation (yes, Zope interfaces implement PEP 246) and runtime adapt hooks
Closing as this is being left out for now.
Reopening this, as duck typing is important according to the BDFL-Delegate (Mark Shannon).
65 remaining items
I'm not sure how you can say that
Iterablealready behaves like a protocol. I tried this code:class C: def __iter__(self) -> Iterator[int]: yield 0 for x in C(): # error: Iterable expected print(x)
This complains that a
Cinstance is not anIterable.Iterablebehaves like a protocol at runtime (it has a customisinstanceoverload).- Ah, cool. So the concerns are more about protocols that *don't* do that, e.g. Sequence or Mapping.
So the concerns are more about protocols that don't do that, e.g. Sequence or Mapping.
Yes, there are four classes
Sequence,Mapping,MutableSequence, andMutableMapping.
Actually, I am thinking maybe we can leave them out from python/typeshed#1220 and consider them later. I have already heard several times that people want mypy to recognize existing runtime protocols (withIterablebeing an absolute winner) so that we could merge python/typeshed#1220 without these four classes soon.- I like that as a first step forward!Reacted by Iddan Aaronsohn
Hi, are there any updates on the remaining collections which are not yet protocols --
Sequence,Mapping,MutableSequence, andMutableMapping?- The plan is to keep them as they are.…On Sun, Apr 22, 2018, 22:30 Chad Dombrova ***@***.***> wrote: Hi, are there any updates on the remaining collections which are not yet protocols -- Sequence, Mapping, MutableSequence, and MutableMapping? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMuvI2i6OCmcBEm16b8ZyLI8mR5tcks5trWbkgaJpZM4CvSAy> .
The plan is to keep them as they are.
What's the reasoning behind that? Even if the abc classes don't become protocols, it would be convenient to have
Protocolvariants of these in typing or mypy_extensions, so that users who wish to treat these as protocols within their code don't have to write their ownProtocolclasses for these common cases.What's the reasoning behind that?
We have no options now. The feature cut-off for Python 3.7 has passed long ago, and this festure would require a runtime change in
collections.abc.Strong -1 on
typing.Mappingandcollections.abc.Mappinghaving different semantics, this will only confuse people.We had a discussion over e-mail with @gvanrossum recently, we are both +0 on making these protocols (only +0 mostly because they are large, while protocols should be more compact), so we can consider this again in one year for Python 3.8.
I have some concerns about turning
Sequenceetc. into protocols. Several of the method signatures in these ABCs are pretty subtle, so it's quite easy to write a class that almost implements, say,Sequence, but not quite because of some signatures being incompatible. With an explicit base class a mismatch will immediately be reported by a type checker -- otherwise it's quite possible to write something that looks like a sequence but actually isn't, and checking that by only reading the code can be hard. This isn't a major problem for most existing protocols intypingsince the signatures are pretty obvious.Also, currently
isinstanceworks with them, so we'd have to support that in the future -- so these protocols would be "runtime" protocols. This brings the risk that the runtime and static views of subtyping become inconsistent because of minor signature differences.Examples of somewhat tricky signatures in
Sequence:@overload # Overloads are a somewhat tricky feature def __getitem__(self, i: int) -> _T_co: ... @overload def __getitem__(self, s: slice) -> Sequence[_T_co]: ... def __contains__(self, x: object) -> bool: ... # object argument type may be unexpected
Reacted by Géry Ogam@JukkaL actually yes, I have seen someone on gitter recently confused by
__contains__.@jakebailey (MS Language Server representative)
PEP 544 is now accepted, mypy (and some other type checkers) have good support for the PEP, all necessary infrastructure updates have been made. So I am finally closing this issue as fixed.
Reacted by Emma Smith and Tetsuya MorimotoAlso, currently
isinstanceworks with them, so we'd have to support that in the future -- so these protocols would be "runtime" protocols. This brings the risk that the runtime and static views of subtyping become inconsistent because of minor signature differences.
In mypy's typing,py there's a neat feature called Protocol. Perhaps we should add this to the PEP. Or perhaps it's similar to a Python ABC?