mypy is unable to assign Callable taking Concatenate to Union of that Callable and another using the same ParamSpec

Bug Report

mypy is unable to assign Callable[Concatenate[B, C], D] to Union[Callable[Concatenate[B, C], D], Callable[C, D].

To Reproduce

from typing import TypeVar, ParamSpec, Concatenate, cast
from collections.abc import Callable

A = TypeVar("A")
B = TypeVar("B")
C = ParamSpec("C")
D = TypeVar("D")


def precall_guard_none(
    a: A, b: B | None, f: Callable[Concatenate[A, B, C], D]
) -> Callable[Concatenate[B, C], D] | Callable[C, D]:
    r: Callable[Concatenate[B, C], D] | Callable[C, D]
    if b is None:

        def g(b: B, *args: C.args, **kwargs: C.kwargs) -> D:
            return f(a, b, *args, **kwargs)

        r = g  # This should be permitted because the type of g is one of the variants of the type of r, a Union
    else:

        def h(*args: C.args, **kwargs: C.kwargs) -> D:
            return f(a, cast(B, b), *args, **kwargs)  # This cast is surprisingly necessary

        r = h

    return r

Expected Behavior

I expect no error.

Actual Behavior

mypy produces this error:

main.py:19: error: Incompatible types in assignment (expression has type "Callable[[B, **C], D]", variable has type "Union[Callable[[B, **C], D], Callable[C, D]]")  [assignment]

Your Environment

mypy playground

Mypy version used: 1.2.0
Mypy command-line flags: --strict
Python version used: 3.11

The problem seems to be with using Unions inside some other type.

I ran into the same issue, but with Dict.

... has incompatible type "Dict[str, str]"; expected "Dict[str, Union[str, bool]]"

@nikochiko

@nikochiko that's an unrelated issue. Mypy is correct here; see https://mypy.readthedocs.io/en/stable/common_issues.html#invariance-vs-covariance. Let's keep this issue focused on the ParamSpec problem.

Ah, TIL. Thanks for the pointer.

I may have the same issue but with Optional

from typing import Concatenate, ParamSpec, Callable, Optional


P = ParamSpec("P")

def func1(func: Callable[Concatenate[str, P], None], *args):
    pass

def func2(func: Optional[Callable[Concatenate[str, P], None]], *args):
    pass

def func3(a: str, b: str):
    pass

func1(func3, "hello")
func2(func3, "hello")

test.py:18: error: Argument 1 to "func2" has incompatible type "Callable[[str, str], Any]"; expected "Optional[Callable[[str, VarArg(<nothing>), KwArg(<nothing>)], None]]"  [arg-type]

func1 works, func2 not.

I have done some research, those are my findings.

The issue seems to be originated here

mypy/mypy/constraints.py

Lines 256 to 288 in 7fe1fdd

    
           # Now the potential subtype is known not to be a Union or a type 
        
           # variable that we are solving for. In that case, for a Union to 
        
           # be a supertype of the potential subtype, some item of the Union 
        
           # must be a supertype of it. 
        
           if direction == SUBTYPE_OF and isinstance(actual, UnionType): 
        
               # If some of items is not a complete type, disregard that. 
        
               items = simplify_away_incomplete_types(actual.items) 
        
               # We infer constraints eagerly -- try to find constraints for a type 
        
               # variable if possible. This seems to help with some real-world 
        
               # use cases. 
        
               return any_constraints( 
        
                   [infer_constraints_if_possible(template, a_item, direction) for a_item in items], 
        
                   eager=True, 
        
               ) 
        
           if direction == SUPERTYPE_OF and isinstance(template, UnionType): 
        
               # When the template is a union, we are okay with leaving some 
        
               # type variables indeterminate. This helps with some special 
        
               # cases, though this isn't very principled. 
        
               result = any_constraints( 
        
                   [ 
        
                       infer_constraints_if_possible(t_item, actual, direction) 
        
                       for t_item in template.items 
        
                   ], 
        
                   eager=False, 
        
               ) 
        
               if result: 
        
                   return result 
        
               elif has_recursive_types(template) and not has_recursive_types(actual): 
        
                   return handle_recursive_union(template, actual, direction) 
        
               return [] 
        
           # Remaining cases are handled by ConstraintBuilderVisitor. 
        
           return template.accept(ConstraintBuilderVisitor(actual, direction))

While for func1 it checks constraints with

mypy/mypy/constraints.py

Line 288 in 7fe1fdd

return template.accept(ConstraintBuilderVisitor(actual, direction))

For func2 it enters here

mypy/mypy/constraints.py

Lines 270 to 285 in 7fe1fdd

    
           if direction == SUPERTYPE_OF and isinstance(template, UnionType): 
        
               # When the template is a union, we are okay with leaving some 
        
               # type variables indeterminate. This helps with some special 
        
               # cases, though this isn't very principled. 
        
               result = any_constraints( 
        
                   [ 
        
                       infer_constraints_if_possible(t_item, actual, direction) 
        
                       for t_item in template.items 
        
                   ], 
        
                   eager=False, 
        
               ) 
        
               if result: 
        
                   return result 
        
               elif has_recursive_types(template) and not has_recursive_types(actual): 
        
                   return handle_recursive_union(template, actual, direction) 
        
               return []

then, in infer_constraints_if_possible, here

mypy/mypy/constraints.py

Lines 301 to 304 in 7fe1fdd

    
           if direction == SUPERTYPE_OF and not mypy.subtypes.is_subtype( 
        
               actual, erase_typevars(template) 
        
           ): 
        
               return None

because the two functions (func1 Callable[[str, str], Any] and Callable[[str, *Any, **Any], None]) are not one the subtypes of the other (I don't know if this is correct but it seems reasonable). In is_subtype the function is checked with

mypy/mypy/subtypes.py

Line 332 in 7fe1fdd

return left.accept(SubtypeVisitor(orig_right, subtype_context, proper_subtype))

To sum up, func1 is checked using ConstraintBuilderVisitor while func2 is checked with SubtypeVisitor

I have tried with this change and it works good, but it seems like an hack to me.

diff --git a/mypy/constraints.py b/mypy/constraints.py
index 33230871b..2de4ea1e9 100644
--- a/mypy/constraints.py
+++ b/mypy/constraints.py
@@ -298,8 +298,10 @@ def infer_constraints_if_possible(
     """
     if direction == SUBTYPE_OF and not mypy.subtypes.is_subtype(erase_typevars(template), actual):
         return None
-    if direction == SUPERTYPE_OF and not mypy.subtypes.is_subtype(
-        actual, erase_typevars(template)
+    if (
+        direction == SUPERTYPE_OF
+        and not isinstance(get_proper_type(template), CallableType)
+        and not mypy.subtypes.is_subtype(actual, erase_typevars(template))
     ):
         return None
     if (

I hope that this would be useful for someone

The original example works on master, likely fixed by #15837

Navigation Menu

Repository navigation

mypy is unable to assign Callable taking Concatenate to Union of that Callable and another using the same ParamSpec #15177

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	# Now the potential subtype is known not to be a Union or a type
	# variable that we are solving for. In that case, for a Union to
	# be a supertype of the potential subtype, some item of the Union
	# must be a supertype of it.
	if direction == SUBTYPE_OF and isinstance(actual, UnionType):
	# If some of items is not a complete type, disregard that.
	items = simplify_away_incomplete_types(actual.items)
	# We infer constraints eagerly -- try to find constraints for a type
	# variable if possible. This seems to help with some real-world
	# use cases.
	return any_constraints(
	[infer_constraints_if_possible(template, a_item, direction) for a_item in items],
	eager=True,
	)
	if direction == SUPERTYPE_OF and isinstance(template, UnionType):
	# When the template is a union, we are okay with leaving some
	# type variables indeterminate. This helps with some special
	# cases, though this isn't very principled.
	result = any_constraints(
	[
	infer_constraints_if_possible(t_item, actual, direction)
	for t_item in template.items
	],
	eager=False,
	)
	if result:
	return result
	elif has_recursive_types(template) and not has_recursive_types(actual):
	return handle_recursive_union(template, actual, direction)
	return []

	# Remaining cases are handled by ConstraintBuilderVisitor.
	return template.accept(ConstraintBuilderVisitor(actual, direction))

	if direction == SUPERTYPE_OF and not mypy.subtypes.is_subtype(
	actual, erase_typevars(template)
	):
	return None

mypy is unable to assign Callable taking Concatenate to Union of that Callable and another using the same ParamSpec #15177

Description

Activity

nikochiko commented on May 3, 2023

JelleZijlstra commented on May 3, 2023

nikochiko commented on May 4, 2023

matperc commented on May 29, 2023

matperc commented on May 29, 2023

ilevkivskyi commented on Aug 16, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions