Skip to content

ParamSpec substitution: difference in behaviour between collections.abc.Callable and typing.Callable #102723

Closed as not planned
Closed as not planned
ParamSpec substitution: difference in behaviour between collections.abc.Callable and typing.Callable#102723
Assignees
sobolevn
Labels
3.12only security fixesstdlibStandard Library Python modules in the Lib/ directorytopic-typingtype-bugAn unexpected behavior, bug, or error

Description

@AlexWaygood
AlexWaygood
opened on Mar 15, 2023

There is currently a difference in behaviour between collections.abc.Callable and typing.Callable for the following edge case involving ParamSpec substitution:

Running PGUpdate|x64 interpreter...
Python 3.12.0a6+ (heads/main:12226bec25, Mar 10 2023, 17:32:23) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import typing as t, collections.abc as c
>>> P, T = t.ParamSpec("P"), t.TypeVar("T")
>>> t.Callable[P, T][[P, str], bool][int]
typing.Callable[[int, str], bool]
>>> c.Callable[P, T][[P, str], bool][int]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen _collections_abc>", line 501, in __getitem__
  File "<frozen _collections_abc>", line 456, in __new__
TypeError: Callable must be used as Callable[[arg, ...], result].

According to PEP-612, this is an invalid substitution, for two reasons:

  1. Parameters can only be prepended to a ParamSpec in a parameters list using typing(_extensions).Concatenate.
  2. Appending parameters to a ParamSpec in a parameters list is disallowed.

As such, the behaviour of collections.abc.Callable is more correct here, so ideally we'd change the behaviour of typing.Callable to match collections.abc.Callable.

However, this error should hopefully be caught by static type checkers anyway, and this is a false negative rather than a false positive. The runtime makes no promises that it will raise TypeError on all invalid substitutions, so fixing this should be low priority, in my opinion. We should first concentrate on fixing substitutions where the runtime raises exceptions, even though it shouldn't. For example:

This discrepancy in behaviour was first uncovered as part of a broader discussion in:

Cc. @sobolevn

Activity

  1. AlexWaygood
    added
    type-bugAn unexpected behavior, bug, or error
    stdlibStandard Library Python modules in the Lib/ directory
    topic-typing
    3.12only security fixes
    on Mar 15, 2023
  2. sobolevn
    self-assigned this
    on Mar 16, 2023

  3. sobolevn commented on Mar 16, 2023

    @sobolevn
    sobolevn
    on Mar 16, 2023
    Member

    I am going to research, how hard it is to change.

    If this is a rather simple patch, I am going to apply it.
    If this is a complex one, I will close this issue as "won't fix".

    Because parameters and their substitution is a very hard thing to get right.

  4. sobolevn commented on Mar 16, 2023

    @sobolevn
    sobolevn
    on Mar 16, 2023
    Member

    Wait, I am confused again. Why c.Callable[P, T][[P, str], bool] is allowed?

    It seems like P can only be substituted with:

    1. Some other ParamSpec variable: P -> P2
    2. ...
    3. Arguments list without any ParamSpec variables: [int, str]
    4. Concatenate with ParamSpec: Concatenate[int, P]

    Am I missing something?

  5. AlexWaygood commented on Mar 16, 2023

    @AlexWaygood
    AlexWaygood
    on Mar 16, 2023
    MemberAuthor

    You're correct, c.Callable[P, T][[P, str], bool] should also, strictly speaking, not be allowed.

    Honestly, I think we should probably leave this one. The runtime allows all sorts of crazy things that should be rejected by type checkers. My personal favourite (we should make this an alias for Any):

    >>> list[r"¯\_(ツ)_/¯"]
list['¯\\_(ツ)_/¯']

    We've never promised that the runtime would disallow all things that should be rejected by type checkers, and I don't want us to start making that promise now.

    The fact that c.Callable[P, T][[P, str], bool][int] raises a TypeError is, strictly speaking, correct -- but I think it's almost certainly just an accident of the current implementation rather than something that was designed. So we shouldn't spend time trying to change the behaviour (since the behaviour is not incorrect), but we also probably shouldn't spend time trying to make typing.Callable match it.

    #88965 seems like a much more important issue to me, since that involves the runtime incorrectly raising a TypeError for valid substitutions.

  6. sobolevn commented on Mar 16, 2023

    @sobolevn
    sobolevn
    on Mar 16, 2023
    Member

    Yes, I agree. We should close this as won't fix. Because the correct behaviour is very complex to achieve. And will probably break other things.

    I am going to research, how hard it is to change.

    There's no point to address a minor issue like this one by the cost of increasing code complexity.

  7. sobolevn
    closed this as not plannedon Mar 16, 2023

  8. AlexWaygood commented on Mar 16, 2023

    @AlexWaygood
    AlexWaygood
    on Mar 16, 2023
    MemberAuthor

    Would you be interested in taking a look at #88965? Serhiy assigned it to himself several months ago, but there's been no activity on the issue since, and it would be great to have it fixed.

  9. sobolevn commented on Mar 16, 2023

    @sobolevn
    sobolevn
    on Mar 16, 2023
    Member

    I will tomorrow!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.12only security fixesstdlibStandard Library Python modules in the Lib/ directorytopic-typingtype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions