Skip to content

Can't resolve generic type from return type of callable passed as (default) argument? #10854

Open
Open
Can't resolve generic type from return type of callable passed as (default) argument?#10854
Labels
bugmypy got something wrongtopic-type-variables

Description

@posita
posita
opened on Jul 21, 2021

UPDATE: Probably a dup of #3737 and #10504.

What I want to do is define a function that takes a callable that returns a generic as an argument where that generic influences the return type of the function. Something like this:

def doit(constructor: Callable[..., T]) -> List[T]:
    ...

I can't seem to figure out a way to get that done, though, especially when I am supplying a default argument for the callable. Here are my test cases:

from typing import Callable, List, Protocol, TypeVar

_T = TypeVar("_T")

# error: Incompatible default for argument "constructor" (default has type "Type[str]", argument has type "Callable[[int], _T]")
def generate(n: int, constructor: Callable[[int], _T] = str) -> List[_T]:
    return [constructor(i) for i in range(n)]

# error: Need type annotation for "res" (hint: "res: List[<type>] = ...")
res = generate(5)  # res should be of type List[str]
assert res == ["0", "1", "2", "3", "4"]
assert generate(5, float) == [0.0, 1.0, 2.0, 3.0, 4.0]

_T_co = TypeVar("_T_co", covariant=True)

class Constructor(Protocol[_T_co]):
    def __call__(self, n: int) -> _T_co:
        ...

def generate2(n: int, constructor: Constructor[_T] = str) -> List[_T]:
    return [constructor(i) for i in range(n)]

# error: Need type annotation for "res2" (hint: "res2: List[<type>] = ...")
res2 = generate2(5, float)  # res2 should be of type List[float]
assert res2 == [0.0, 1.0, 2.0, 3.0, 4.0]

Runtime:

% python --version
Python 3.9.6
% python test.py
% mypy --version
mypy 0.910
% mypy --config-file=/dev/null test.py
/dev/null: No [mypy] section in config file
test.py:6: error: Incompatible default for argument "constructor" (default has type "Type[str]", argument has type "Callable[[int], _T]")
test.py:10: error: Need type annotation for "res" (hint: "res: List[<type>] = ...")
test.py:24: error: Need type annotation for "res2" (hint: "res2: List[<type>] = ...")
Found 3 errors in 1 file (checked 1 source file)

I tried searching for a similar issue, but was unable to find what I was looking for.

Activity

  1. posita
    added
    bugmypy got something wrong
    on Jul 21, 2021

  2. erictraut commented on Jul 21, 2021

    @erictraut
    erictraut
    on Jul 21, 2021

    I think that your first sample (generate) should work. That looks like a bug in mypy. It works fine in pyright.

    For your second sample (generate2 with the Constructor protocol), the problem is that you've specified that the __call__ method must have a parameter with the name n, but the constructors for str and float don't have a parameter with that name. If you change the __call__ method to indicate that it accepts position-only parameters, it will type check without errors in both mypy and pyright. The error message provided by mypy in this case is not very useful in diagnosing the problem.

        def __call__(self, n: int, /) -> _T_co:

  3. posita commented on Jul 21, 2021

    @posita
    posita
    on Jul 21, 2021
    ContributorAuthor

    Thanks @erictraut! I can verify that your suggestion fixed my simple test case. Here's one closer to the real thing (note the generic in the more complex return type):

    from fractions import Fraction
from typing import List, Protocol, Tuple, TypeVar

_T = TypeVar("_T")
_T_co = TypeVar("_T_co", covariant=True)

class _RationalP(Protocol[_T_co]):
    def __call__(self, numerator: int, denominator: int, /) -> _T_co:
        ...

def generate3(n: int, constructor: _RationalP[_T] = Fraction) -> List[Tuple[float, _T]]:
    return [(float(i), constructor(i, 2)) for i in range(n)]   # ^^^^^^^^^^^^^^^^^^**^^

# error: Need type annotation for "res3"
res3 = generate3(5)  # res3 should be of type List[Tuple[float, Fraction]]
res3_list = list(res3)
print(res3_list)
assert res3_list == [
    (0.0, Fraction(0, 1)),
    (1.0, Fraction(1, 2)),
    (2.0, Fraction(1, 1)),
    (3.0, Fraction(3, 2)),
    (4.0, Fraction(2, 1)),
]

    Runtime:

    % python --version
Python 3.9.6
% python test.py
[(0.0, Fraction(0, 1)), (1.0, Fraction(1, 2)), (2.0, Fraction(1, 1)), (3.0, Fraction(3, 2)), (4.0, Fraction(2, 1))]
% mypy --version
mypy 0.910
% mypy --config-file=/dev/null test.py
/dev/null: No [mypy] section in config file
test.py:15: error: Need type annotation for "res3"
Found 1 error in 1 file (checked 1 source file)

    This doesn't work (at least with MyPy):

    # …
_PairT = Tuple[float, _T]
# …
def generate3(n: int, constructor: _RationalP[_T] = Fraction) -> List[_PairT[_T]]:
    ...
# …

    This gives a new error:

    # …
class _PairT(Tuple[float, _T]):
    ...
# …
# error: Generic tuple types not supported
def generate3(n: int, constructor: _RationalP[_T] = Fraction) -> List[_PairT[_T]]:
    ...
# …

  4. erictraut commented on Jul 22, 2021

    @erictraut
    erictraut
    on Jul 22, 2021

    That should type check without errors. Pyright is fine with it.

    If you explicitly pass Fraction as a second argument to generate3, mypy is happy with it. So this appears to be a bug in mypy's handling of default parameter values.

  5. posita
    changed the title [-]Can't resolve generic type from return type of callable passed as argument?[/-] [+]Can't resolve generic type from return type of callable passed as (default) argument?[/+] on Jul 22, 2021

  6. posita commented on Jul 22, 2021

    @posita
    posita
    on Jul 22, 2021
    ContributorAuthor

    Whoo boy.

    from fractions import Fraction
from typing import List, Protocol, Tuple, TypeVar, overload

_T = TypeVar("_T")
_T_co = TypeVar("_T_co", covariant=True)

class _RationalP(Protocol[_T_co]):
    def __call__(self, numerator: int, denominator: int, /) -> _T_co:
        ...

@overload
def generate4(n: int) -> List[Tuple[float, Fraction]]:
    ...

@overload
def generate4(n: int, constructor: _RationalP[_T]) -> List[Tuple[float, _T]]:
    ...

def generate4(n: int, constructor: _RationalP[_T] = Fraction) -> List[Tuple[float, _T]]:
    return [(float(i), constructor(i, 2)) for i in range(n)]

res4 = generate4(5)  # res4 should be of type Iterator[Tuple[float, Fraction]]
res4_list = list(res4)
print(res4_list)
assert res4_list == [
    (0.0, Fraction(0, 1)),
    (1.0, Fraction(1, 2)),
    (2.0, Fraction(1, 1)),
    (3.0, Fraction(3, 2)),
    (4.0, Fraction(2, 1)),
]

    ☝️ That is some work-around!

    Thanks for all your help narrowing this down!

  7. hauntsaninja commented on Jul 22, 2021

    @hauntsaninja
    hauntsaninja
    on Jul 22, 2021
    Collaborator

    Linking #3737

  8. posita commented on Jul 22, 2021

    @posita
    posita
    on Jul 22, 2021
    ContributorAuthor

    Thanks @hauntsaninja! It does look like this has overlap with #3737, and is probably even a dup, since removing the default argument works, even for the non-protocol version of my simple test:

    from typing import Callable, List, TypeVar

_T = TypeVar("_T")

def generate(n: int, constructor: Callable[[int], _T]) -> List[_T]:
    return [constructor(i) for i in range(n)]

res = generate(5, str)
assert res == ["0", "1", "2", "3", "4"]
assert generate(5, float) == [0.0, 1.0, 2.0, 3.0, 4.0]

    ☝️ No errors with that one.

    In any event, this definitely feels like a dupe of #10504, which I just now found (via #3737).

  9. JelleZijlstra
    added
    topic-type-variables
    on Mar 19, 2022

  10. posita commented on Aug 2, 2023

    @posita
    posita
    on Aug 2, 2023
    ContributorAuthor

    Somewhere between 0.993 and 1.4.1 (yes, I know that's a big range), the work-around identified in #10854 (comment) no longer works. 😞 It results in:

    ...: error: Incompatible default for argument "constructor" (default has type "type[Fraction]", argument has type "_RationalP[_T]")  [assignment]
...: note: "_RationalP[_T].__call__" has type "Callable[[int, int], _T]"

  11. posita commented on Aug 3, 2023

    @posita
    posita
    on Aug 3, 2023
    ContributorAuthor

    New work-around (using None for the default argument and overload to narrow the return type when omitted):

    from fractions import Fraction
from typing import Callable, TypeVar, overload

_T = TypeVar("_T")

@overload
def generate(
    n: int,
) -> list[tuple[float, Fraction]]:
    pass

@overload
def generate(
    n: int,
    constructor: Callable[[int, int], _T],
) -> list[tuple[float, _T]]:
    pass

def generate(
    n: int,
    constructor: Callable[[int, int], _T] | None = None,
) -> list[tuple[float, _T]]:
    if constructor is None:
        # This is a persistent error, but it's limited to the implementation. The interface remains usable.
        constructor = Fraction  # type: ignore
        assert constructor is not None

    return [(float(i), constructor(i, 2)) for i in range(n)]

res1 = generate(5, Fraction)
reveal_type(res1)  # list[tuple[float, Fraction]]

res2 = generate(5, lambda n, d: n / d)
reveal_type(res2)  # list[tuple[float, float]]

res3 = generate(5)
reveal_type(res3)  # list[tuple[float, Fraction]]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugmypy got something wrongtopic-type-variables

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions