Incorrect handling of overloads with defaultdict and MutableMapping

Bug Report
This came from a PR with pandas-stubs. The overloads with defaultdict and MutableMapping aren't working correctly.

Requires a stub file and a test file.

To Reproduce

File defdictover.pyi :

from collections import defaultdict
from typing import TypeVar, MutableMapping, overload, Literal, Hashable, Any

_T_MUTABLE_MAPPING = TypeVar("_T_MUTABLE_MAPPING", bound=MutableMapping, covariant=True)

@overload
def to_dict(
    orient: Literal["records"],
    *,
    into: type[dict] = ...,
) -> list[dict[Hashable, Any]]: ...
@overload
def to_dict(
    orient: Literal["index"],
    *,
    into: defaultdict,
) -> defaultdict[Hashable, dict[Hashable, Any]]: ...
@overload
def to_dict(
    orient: Literal["index"],
    *,
    into: MutableMapping | type[MutableMapping],
) -> MutableMapping[Hashable, dict[Hashable, Any]]: ...

File tstdefdictover.py :

from collections import defaultdict
from typing import Any
from defdictover import to_dict

target: defaultdict[Any, list] = defaultdict(list)

reveal_type(to_dict("index", into=target))

Then mypy tstdefdictover.py produces:

tstdefdictover.py:7: note: Revealed type is "Any"

Expected Behavior

With pyright, we get:

tstdefdictover.py:7:13 - information: Type of "to_dict("index", into=target)" is "defaultdict[Hashable, dict[Hashable, Any]]"

Actual Behavior

See above - it sees the type as Any

Your Environment

Mypy version used: 1.17
Mypy command-line flags: None
Mypy configuration options from mypy.ini (and other config files): None
Python version used: 3.12

Smaller reproducer:

from typing import overload, Any

class A[T]: ...
class B[T](A[T]): ...

@overload
def foo(x: B[Any]) -> B[Any]: ...
@overload
def foo(x: A[Any]) -> A[Any]: ...
def foo(x: A[Any]) -> A[Any]: raise NotImplementedError

b: B[Any]
reveal_type(foo(b))  # N: Revealed type is "Any"

edit: only the Any at the callsite appears to be relevant:

from typing import overload, Any

class A[T]: ...
class B[T](A[T]): ...

@overload
def foo(x: B[int]) -> B[int]: ...
@overload
def foo(x: A[int]) -> A[int]: ...
def foo(x: A[int]) -> A[int]: raise NotImplementedError

b: B[Any]
reveal_type(foo(b))  # N: Revealed type is "Any"

(nevermind about the crash - that was due to an unrelated issue caused by a typo in the reproducer. I'll open a separate issue for it. edit: #19526)

@brianschubert

Is this really a bug? While this specific example seems unambiguous, consider the following:

from typing import overload, Any

class A[T]: ...
class B[T](A[T]): ...

@overload
def foo(x: B[int]) -> B[int]: ...
@overload
def foo(x: A[str]) -> A[str]: ...
def foo(x: A[int] | A[str]) -> A[int] | A[str]: raise NotImplementedError

b: B[Any]
reveal_type(foo(b))  # N: Revealed type is "???"

What should ??? be? In my mind it should be Any as "idk, have your any" fallback. There is no way I could expect B[int] there. I can also understand inferring union (A[str] | B[int]), join (A[int | str]) or A[Any] (the latter with a bit of a stretch), but picking the first overload in such case (where it's Any what decides which overload to pick) is unreasonable IMO.

I don't have enough time to dig the overloads spec now (and I still remember how controversial that page was), but I prefer mypy behavior (Any) over pyright behavior (B[int]) even if that means nonconformance.

Any in the last reproducer by @brianschubert "causes overload ambiguity", we look for such conditions explicitly:

mypy/mypy/checkexpr.py

Lines 6598 to 6653 in af92439

    
           def any_causes_overload_ambiguity( 
        
               items: list[CallableType], 
        
               return_types: list[Type], 
        
               arg_types: list[Type], 
        
               arg_kinds: list[ArgKind], 
        
               arg_names: Sequence[str | None] | None, 
        
           ) -> bool: 
        
               """May an argument containing 'Any' cause ambiguous result type on call to overloaded function? 
        
               Note that this sometimes returns True even if there is no ambiguity, since a correct 
        
               implementation would be complex (and the call would be imprecisely typed due to Any 
        
               types anyway). 
        
               Args: 
        
                   items: Overload items matching the actual arguments 
        
                   arg_types: Actual argument types 
        
                   arg_kinds: Actual argument kinds 
        
                   arg_names: Actual argument names 
        
               """ 
        
               if all_same_types(return_types): 
        
                   return False 
        
               actual_to_formal = [ 
        
                   map_formals_to_actuals( 
        
                       arg_kinds, arg_names, item.arg_kinds, item.arg_names, lambda i: arg_types[i] 
        
                   ) 
        
                   for item in items 
        
               ] 
        
               for arg_idx, arg_type in enumerate(arg_types): 
        
                   # We ignore Anys in type object callables as ambiguity 
        
                   # creators, since that can lead to falsely claiming ambiguity 
        
                   # for overloads between Type and Callable. 
        
                   if has_any_type(arg_type, ignore_in_type_obj=True): 
        
                       matching_formals_unfiltered = [ 
        
                           (item_idx, lookup[arg_idx]) 
        
                           for item_idx, lookup in enumerate(actual_to_formal) 
        
                           if lookup[arg_idx] 
        
                       ] 
        
                       matching_returns = [] 
        
                       matching_formals = [] 
        
                       for item_idx, formals in matching_formals_unfiltered: 
        
                           matched_callable = items[item_idx] 
        
                           matching_returns.append(matched_callable.ret_type) 
        
                           # Note: if an actual maps to multiple formals of differing types within 
        
                           # a single callable, then we know at least one of those formals must be 
        
                           # a different type then the formal(s) in some other callable. 
        
                           # So it's safe to just append everything to the same list. 
        
                           for formal in formals: 
        
                               matching_formals.append(matched_callable.arg_types[formal]) 
        
                       if not all_same_types(matching_formals) and not all_same_types(matching_returns): 
        
                           # Any maps to multiple different types, and the return types of these items differ. 
        
                           return True 
        
               return False

Is this really a bug?

I think mypy is working correctly here and is in compliance with the overload call evaluation algorithm defined in the typing spec. Pyright is nonconformant in this case. This is a known issue in pyright.

@erictraut

Is this really a bug?

I think mypy is working correctly here and is in compliance with the overload call evaluation algorithm defined in the typing spec. Pyright is nonconformant in this case. This is a known issue in pyright.

@erictraut which example are you referring to? The original one I posted, or the second example: #19525 (comment)

which example are you referring to

Both. The second example is a simplified version of the first.

Both. The second example is a simplified version of the first.

Could you elaborate why the first example is correct? In my opinion, inserting b: B[Any] into foo should be evaluated by the first overload

@overload
def foo(x: B[Any]) -> B[Any]: ...

and should give the type B[Any].

Any should be seen as a standin for some unknown static type (a "materialization"; see the spec: https://typing.python.org/en/latest/spec/concepts.html#materialization ). This example uses a number of distinct Anys, and in general we have to infer a type that works for all possible materializations. In this case, there are many possible materializations that would make it so the call with B[Any] as an argument hits the second overload. For example, the Any in the argument and in the parameter type of the second overload could materialize to one type (e.g., int), while the Any in the parameter type of the first overload could materialize to another type (e.g., str).

This means that we hit step 5 of the overload evaluation spec and there are multiple matching overloads with distinct return types. Therefore, we return Any.

fix: constraint instead of bound to get rid of Any python/mypy#19525 …

Any should be seen as a standin for some unknown static type (a "materialization"; see the spec: https://typing.python.org/en/latest/spec/concepts.html#materialization ). This example uses a number of distinct Anys, and in general we have to infer a type that works for all possible materializations. In this case, there are many possible materializations that would make it so the call with B[Any] as an argument hits the second overload. For example, the Any in the argument and in the parameter type of the second overload could materialize to one type (e.g., int), while the Any in the parameter type of the first overload could materialize to another type (e.g., str).

This means that we hit step 5 of the overload evaluation spec and there are multiple matching overloads with distinct return types. Therefore, we return Any.

Can you make the same analysis with my original example involving defaultdict and MutableMapping, because I'm having a hard time seeing how your analysis applies in that case.

feat(index): append (#1282)

Your original example had defaultdict (which means defaultdict[Any, Any]) and MutableMapping (i.e., MutableMapping[Any, Any]), so it's closely analogous. Unfortunately the type system doesn't contain a type that represents all kinds of defaultdicts, which is perhaps what you'd want here.

Since we have identified the root cause and realized that this behavior is intended and even spec-compliant, I'm closing this ticket as "not planned" - there's no bug on the mypy side.

feat(index): append (pandas-dev#1282)

Navigation Menu

Repository navigation

Incorrect handling of overloads with `defaultdict` and `MutableMapping` #19525

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	def any_causes_overload_ambiguity(
	items: list[CallableType],
	return_types: list[Type],
	arg_types: list[Type],
	arg_kinds: list[ArgKind],
	arg_names: Sequence[str \| None] \| None,
	) -> bool:
	"""May an argument containing 'Any' cause ambiguous result type on call to overloaded function?

	Note that this sometimes returns True even if there is no ambiguity, since a correct
	implementation would be complex (and the call would be imprecisely typed due to Any
	types anyway).

	Args:
	items: Overload items matching the actual arguments
	arg_types: Actual argument types
	arg_kinds: Actual argument kinds
	arg_names: Actual argument names
	"""
	if all_same_types(return_types):
	return False

	actual_to_formal = [
	map_formals_to_actuals(
	arg_kinds, arg_names, item.arg_kinds, item.arg_names, lambda i: arg_types[i]
	)
	for item in items
	]

	for arg_idx, arg_type in enumerate(arg_types):
	# We ignore Anys in type object callables as ambiguity
	# creators, since that can lead to falsely claiming ambiguity
	# for overloads between Type and Callable.
	if has_any_type(arg_type, ignore_in_type_obj=True):
	matching_formals_unfiltered = [
	(item_idx, lookup[arg_idx])
	for item_idx, lookup in enumerate(actual_to_formal)
	if lookup[arg_idx]
	]

	matching_returns = []
	matching_formals = []
	for item_idx, formals in matching_formals_unfiltered:
	matched_callable = items[item_idx]
	matching_returns.append(matched_callable.ret_type)

	# Note: if an actual maps to multiple formals of differing types within
	# a single callable, then we know at least one of those formals must be
	# a different type then the formal(s) in some other callable.
	# So it's safe to just append everything to the same list.
	for formal in formals:
	matching_formals.append(matched_callable.arg_types[formal])
	if not all_same_types(matching_formals) and not all_same_types(matching_returns):
	# Any maps to multiple different types, and the return types of these items differ.
	return True
	return False

Incorrect handling of overloads with defaultdict and MutableMapping #19525

Description

Activity

brianschubert commented on Jul 28, 2025

brianschubert commented on Jul 28, 2025

sterliakov commented on Jul 29, 2025

erictraut commented on Jul 29, 2025

Dr-Irv commented on Jul 29, 2025

erictraut commented on Jul 30, 2025

cmp0xff commented on Jul 30, 2025

JelleZijlstra commented on Jul 30, 2025

Dr-Irv commented on Jul 30, 2025

JelleZijlstra commented on Aug 1, 2025

sterliakov commented on Aug 8, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Incorrect handling of overloads with `defaultdict` and `MutableMapping` #19525