Cannot parameterise typing_extensions.Protocol with a ParamSpec or TypeVarTuple on py310+

@Tenzer

This change seems to have broken the latest version of pydash when used with typing-extensions 4.6.0. There's a bug report about this here: dgilland/pydash#197.

I did a bisect of every change between typing-extensions 4.5.0 and 4.6.0, and this was pointed out as the culprit. I've had a look at the code changes here but don't have a good enough grasp of what's going wrong that is causing this to fail.

This seems to be the relevant Pydash code that relates to the error:
https://github.com/dgilland/pydash/blob/051fe69c3e523f903a0a0ea6ca6a5c2d4b83a3e7/src/pydash/utilities.py#L577-L638

Originally posted by @Tenzer in #137 (comment)

My guess is that this change might be the culprit, but haven't checked that at all: https://github.com/python/typing_extensions/pull/137/files#r1164017024

Minimal repro:

>>> from typing_extensions import Protocol, ParamSpec
>>> P = ParamSpec("P")
>>> class Foo(Protocol[P]): ...
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\alexw\AppData\Local\Programs\Python\Python311\Lib\typing.py", line 344, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\typing_extensions\src\typing_extensions.py", line 672, in __class_getitem__
    raise TypeError(
TypeError: Parameters to Protocol[...] must all be type variables. Parameter 1 is ~P

Works fine on typing-extensions==4.5.0 (with Python 3.7 and Python 3.10), and also works fine if you use typing.{Protocol,ParamSpec}. So, definitely looks like a typing_extensions bug.

Interestingly, the bug only reproduces if you use Python >=3.10 (i.e., it has to be a version of Python where typing.ParamSpec is a thing for the bug to be reproducible).

Seems like the fundamental issue is that:

We reimplement the logic from typing.Generic.__class_getitem__ in typing_extensions.Protocol.__class_getitem__. (Ideally we wouldn't have to do this, but the typing module on Python 3.7 doesn't let us have typing_extensions.Protocol subclass typing.Generic.)
The implementation of typing.Generic.__class_getitem__ changed on Python 3.10 and Python 3.11, but we haven't reflected those changes in typing_extensions.
Prior to typing_extensions==4.6.0, we only re-implemented Protocol on Python <=3.7, but now we re-implement Protocol on <=3.11. This means that the latent bug has been exposed on Python 3.10 and 3.11, hence the breakage.

Not that anybody asked, but the bug also means that you can't parameterise a typing_extensions.Protocol with a TypeVarTuple on 3.10/3.11:

>>> from typing_extensions import *
>>> Ts = TypeVarTuple("Ts")
>>> class Foo(Protocol[*Ts]): ...
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\alexw\AppData\Local\Programs\Python\Python311\Lib\typing.py", line 344, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\typing_extensions\src\typing_extensions.py", line 672, in __class_getitem__
    raise TypeError(
TypeError: Parameters to Protocol[...] must all be type variables. Parameter 1 is *Ts

@AlexWaygood

Thanks @AlexWaygood for investigating! Are you working on a fix?

I am looking into several potential fixes...

As usual with typing_extensions, there are several complications to grapple with :)

I also have a busy day at work, so might not be able to look at this properly until this evening!

No hurry, I can give it a try later today too.

We'll probably want to release a bugfix 4.6.1 with this fix.

One possible fix involves copying over the logic for Generic.__class_getitem__ from 3.12a7 to typing_extensions.Protocol.__class_getitem__. In order to do that cleanly, though, we might have to first backport some unrelated fixes that were made to Unpack in 3.11, which comes with its own risks.

I looked into whether we could instead inherit from typing.Generic on 3.8+, which would mean we wouldn't have to have a Protocol.__class_getitem__ on 3.8+. I'm still investigating that, but suffice it to say that there are complications to that approach as well.

Just leaving this here for posterity:

They're probably mostly referring to #179, not this issue.

Possible, #158 would probably have been a more fitting issue for the screenshot. But I claim innocence in any case. :)

@JelleZijlstra

Oh, I'm definitely guilty for this bug, if we're pointing fingers 😅

I agree with @JelleZijlstra though, there seems to be far more breakage from pydantic/pydantic#5821 than from this issue!

I am looking into several potential fixes...

As usual with typing_extensions, there are several complications to grapple with :)

I also have a busy day at work, so might not be able to look at this properly until this evening!

I have a fix in the works now.

Navigation Menu

Repository navigation

Cannot parameterise `typing_extensions.Protocol` with a `ParamSpec` or `TypeVarTuple` on py310+ #181

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Cannot parameterise typing_extensions.Protocol with a ParamSpec or TypeVarTuple on py310+ #181

Description

Activity

AlexWaygood commented on May 23, 2023

AlexWaygood commented on May 23, 2023

AlexWaygood commented on May 23, 2023

AlexWaygood commented on May 23, 2023

AlexWaygood commented on May 23, 2023

JelleZijlstra commented on May 23, 2023

AlexWaygood commented on May 23, 2023

JelleZijlstra commented on May 23, 2023

AlexWaygood commented on May 23, 2023

srittau commented on May 23, 2023

JelleZijlstra commented on May 23, 2023

srittau commented on May 23, 2023

AlexWaygood commented on May 23, 2023

AlexWaygood commented on May 23, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Cannot parameterise `typing_extensions.Protocol` with a `ParamSpec` or `TypeVarTuple` on py310+ #181