[1.14 regression] Unpacking an iterator converts the inner type to Any

Bug Report

Unpacking an iterator into a list produces a list[Any] instead of the more specific type. This worked correctly in mypy 1.13.0 but no longer works in 1.14.0.

To Reproduce

from typing import reveal_type
x = [1, 2]
reveal_type([*reversed(x)])
y = [3, 4]
reveal_type([*map(str, y)])

Expected Behavior

In mypy 1.13.0:

x.py:3: note: Revealed type is "builtins.list[builtins.int]"
x.py:5: note: Revealed type is "builtins.list[builtins.str]"

Actual Behavior

In mypy 1.14.0:

x.py:3: note: Revealed type is "builtins.list[Any]"
x.py:5: note: Revealed type is "builtins.list[Any]"

Your Environment

Mypy version used: 1.14.0
Mypy command-line flags: none
Mypy configuration options from mypy.ini (and other config files): none
Python version used: 3.13.0

Bisects to this typeshed sync: #18057

@tungol

Looks like this is python/typeshed#12851 ? cc @tungol That change probably makes type checkers a little slower too

I'll take a look

Using the reproduction above, I stepped through. The problem comes when we get into mypy.maptype

Stack starting at the reveal expression:

  mypy/mypy/checkexpr.py(4681)visit_reveal_expr()
-> revealed_type = self.accept(
  mypy/mypy/checkexpr.py(5903)accept()
-> typ = node.accept(self)
  mypy/mypy/nodes.py(2313)accept()
-> return visitor.visit_list_expr(self)
  mypy/mypy/checkexpr.py(4991)visit_list_expr()
-> return self.check_lst_expr(e, "builtins.list", "<list>")
  mypy/mypy/checkexpr.py(5057)check_lst_expr()
-> out = self.check_call(
  mypy/mypy/checkexpr.py(1571)check_call()
-> return self.check_callable_call(
  mypy/mypy/checkexpr.py(1757)check_callable_call()
-> callee = self.infer_function_type_arguments(
  mypy/mypy/checkexpr.py(2088)infer_function_type_arguments()
-> inferred_args, _ = infer_function_type_arguments(
  mypy/mypy/infer.py(56)infer_function_type_arguments()
-> constraints = infer_constraints_for_callable(
  mypy/mypy/constraints.py(243)infer_constraints_for_callable()
-> actual_type = mapper.expand_actual_type(
  mypy/mypy/argmap.py(204)expand_actual_type()
-> return map_instance_to_supertype(
  mypy/mypy/maptype.py(42)map_instance_to_supertype()
-> return map_instance_to_supertypes(instance, superclass)[0]
> mypy/mypy/maptype.py(49)map_instance_to_supertypes()
-> for path in class_derivation_paths(instance.type, supertype):

Where instance = builtins.reversed[builtins.int] and supertype = <TypeInfo typing.Iterable>.

Up the stack a couple places, at argmap.py(204)expand_actual_type(), we successfully pass a check that is_subtype(actual_type, self.context.iterable_type) in both cases, but maptype.map_instance_to_supertype says in its docstring that If 'superclass' is not a nominal superclass of 'instance.type', then all type arguments are mapped to 'Any'.

So we reach maptype.py(49)map_instance_to_supertypes() and when reversed is an explicit subclass of Iterator we run class_derivation_paths(instance.type, supertype) and get [[<TypeInfo typing.Iterator>, <TypeInfo typing.Iterable>]], but when it isn't an explicit subclass of Iterator we hit the same line and get [] instead and end up returning Any.

The iterable builtins are the most prominent and most important, but I'd expect that this issue would come up for any iterator type that doesn't explicitly inherit from Iterator or Iterable, so this situation would ideally be properly handled by mypy without needing to inherit explicitly from those.

I can make a point release to fix this since it seems pretty bad. If it's easier/faster we could just revert the typeshed change (python/typeshed#12851 -- but maybe the others mentioned in the description have similar bugs?) in mypy for 1.14.1 while we make a proper fix for the 1.15 release.

Yes, there's a similar bug for the others mentioned in the description, along with anything that has an __iter__ method but doesn't explicitly inherit from Iterable. This code reproduces the issue for all of the iterable builtins as well as a couple things from itertools. Some of the other iterables from that series of typeshed MRs need a little more effort to construct and I didn't bother here, but I have no reason to think they're any different:

import itertools
from typing import reveal_type

x = [1, 2]
y = [3, 4]
reveal_type([*enumerate(x)])
reveal_type([*filter(lambda x: True, x)])
reveal_type([*map(str, x)])
reveal_type([*reversed(x)])
reveal_type([*zip(x, y)])

reveal_type([*itertools.cycle(x)])
reveal_type([*itertools.permutations(x)])

1.13.0 says:

test.py:6: note: Revealed type is "builtins.list[tuple[builtins.int, builtins.int]]"
test.py:7: note: Revealed type is "builtins.list[builtins.int]"
test.py:8: note: Revealed type is "builtins.list[builtins.str]"
test.py:9: note: Revealed type is "builtins.list[builtins.int]"
test.py:10: note: Revealed type is "builtins.list[tuple[builtins.int, builtins.int]]"
test.py:12: note: Revealed type is "builtins.list[builtins.int]"
test.py:13: note: Revealed type is "builtins.list[builtins.tuple[builtins.int, ...]]"

1.14.0 says:

test.py:6: note: Revealed type is "builtins.list[Any]"
test.py:7: note: Revealed type is "builtins.list[Any]"
test.py:8: note: Revealed type is "builtins.list[Any]"
test.py:9: note: Revealed type is "builtins.list[Any]"
test.py:10: note: Revealed type is "builtins.list[Any]"
test.py:12: note: Revealed type is "builtins.list[Any]"
test.py:13: note: Revealed type is "builtins.list[Any]"

This code reproduces the issue on any version of mypy, not just 1.14.0:

from typing import reveal_type, Iterable, Iterator, TypeVar, Generic

_T = TypeVar("_T")

class Iter1(Iterable[_T]):
    def __init__(self, value: list[_T]) -> None:
        self.value = value

    def __iter__(self) -> Iterator[_T]:
        return iter(self.value)

class Iter2(Generic[_T]):
    def __init__(self, value: list[_T]) -> None:
        self.value = value

    def __iter__(self) -> Iterator[_T]:
        return iter(self.value)

x = [1, 2]
reveal_type([*Iter1(x)])  # Revealed type is "builtins.list[builtins.int]"
reveal_type([*Iter2(x)])  # Revealed type is "builtins.list[Any]"

If there's an equivalent of map_instance_to_supertype which handles structural super/sub classes already in the codebase somewhere that would be an easy fix. Otherwise I don't think I understand what mypy.argmap and mypy.maptype are doing well enough to take a stab at fixing the underlying cause myself.

Reverting the typeshed change(s) as a quick fix probably makes sense. I'd probably go with at least the builtins and the itertools, but at the high end, really covering everything would mean making sure that everything in typeshed with an __iter__ method has nominal inheritance Iterable, not just structural inheritance.

I looked at everything in typeshed/stdlib that has an __iter__ method on it; here's my analysis. Note that I didn't check the actual behavior. I just looked at the inheritance in the stubs.

_asyncio.Future: Affected
_contextvars.Context: Not affected; inherits from Mapping
_csv.Reader: Affected
_csv._reader: Affected
_ctypes.Array: Affected
_io._IOBase: Affected
_io._TextIOBase: Affected, but inherits from _io._IOBase and will be fixed by that fix.
_typeshed.SupportsIter: Affected, but it's a Protocol class, and also from _typeshed. Should not be changed.
_typeshed.wsgi.InputStream: Affected, but it's a Protocol class, and also from _typeshed. Should not be changed.
_weakrefset.WeakSet: Not affected; inherits from MutableSet
asyncio.locks._ContextManagerMixin: Affected (This looks like a weird class and maybe it should be broken here?)
builtins.str: Not affected; inherits from Sequence
builtins.bytes: Not affected; inherits from Sequence
builtins.bytearray: Not affected; inherits from MutableSequence
builtins.memoryview: Not affected; inherits from Sequence
builtins.tuple: Not affected; inherits from Sequence
builtins.list: Not affected; inherits from MutableSequence
builtins.dict: Not affected; inherits from MutableMapping
builtins.set: Not affected; inherits from MutableSet
builtins.frozenset: Not affected; inherits from AbstractSet
builtins.enumerate: Affected
builtins.range: Not affected; inherits from Sequence
builtins.filter: Affected
builtins.map: Affected
builtins.reversed: Affected
builtins.zip: Affected
cgi.FieldStorage: Affected
codecs.StreamRecoder: Affected but inherits from typing.BinaryIO
codecs.StreamReader: Affected
codecs.StreamReaderWriter: Affected but inherits from typing.TextIO
collections.UserDict: Not affected; inherits from MutableMapping
collections.UserString: Not affected; inherits from Sequence
collections.ChainMap: Not affected; inherits from MutableMapping
configparser.RawConfigParser: Not affected; inherits from MutableMapping
configparser.SectionProxy: Not affected; inherits from MutableMapping
configparser.ConverterMapping: Not affected; inherits from MutableMapping
csv.DictReader: Affected
dbm._Database: Not affected; inherits from MutableMapping
dbm.dumb._Database: Not affected; inherits from MutableMapping
dbm.sqlite3._Database: Not affected; inherits from MutableMapping
dis.Bytecode: Affected
email.message.Message: Affected
enum.EnumMeta: Affected
enum.Flag: Affected
fileinput.FileInput: Affected
http.client.HTTPResponse: Affected but inherits from typing.BinaryIO
http.cookiejar.CookieJar: Affected
importlib.metadata.EntryPoint: Partially affected; It has different inheritance on different python versions
importlib.metadata.Deprecated: Affected
importlib.metadata._meta.PackageMetadata: Affected, but it's a Protocol class
ipaddress._BaseNetwork: Affected
itertools.count: Affected
itertools.cycle: Affected
itertools.repeat: Affected
itertools.accumulate: Affected
itertools.chain: Affected
itertools.compress: Affected
itertools.dropwhile: Affected
itertools.filterfalse: Affected
itertools.groupby: Affected
itertools.isslice: Affected
itertools.starmap: Affected
itertools.takewhile: Affected
itertools.zip_longest: Affected
itertools.product: Affected
itertools.permutations: Affected
itertools.combinations: Affected
itertools.combinations_with_replacement: Affected
itertools.pairwise: Affected
itertools.batched: Affected
mailbox.Mailbox: Affected
mailbox._ProxyFile: Affected
mmap.mmap: Affected
multiprocessing.managers._BaseDictProxy:  Not affected; inherits from MutableMapping
multiprocessing.managers.DictProxy:  Not affected; inherits from MutableMapping
multiprocessing.pool.IMapIterator: Affected
os._Environ:  Not affected; inherits from MutableMapping
os._ScandirIterator: Affected
os._wrap_close: Affected
shelve.Shelf:  Not affected; inherits from MutableMapping
shlex.shlex: Affected
sqlite3.Cursor: Affected
sqlite3.Row: Not affected; inherits from Sequence
tarfile.Tarfile: Affected
tempfile._TemporaryFileWrapper: Affected but inherits from typing.IO
tempfile.SpooledTemporaryFile: Affected but inherits from typing.IO
traceback.FrameSummary: Affected
types.MappingProxyType: Not affected; inherits from Mapping
types.GeneratorType: Not affected; inherits from Generator
typing.TypeVarTuple: Affected
typing.Iterable: Not affected by definition
typing.Iterator: Not affected by definition
typing.Generator: Not affected; inherits from Iterator
typing.Sequence: Not affected; inherits from Reversible and Collection
typing.ItemsView: Not affected; inherits from AbstractSet
typing.KeysView: Not affected; inherits from AbstractSet
typing.ValuesView: Not affected; inherits from Collection
typing.IO: Affected
typing_extensions.TypeVarTuple: Affected
unittest.suite.BaseTestSuite: Affected
weakref.WeakValueDictionary: Not affected; inherits from MutableMapping
weakref.WeakKeyDictionary: Not affected; inherits from MutableMapping
wsgiref.types.InputStream: Affected, but it's a Protocol class
wsgiref.util.FileWrapper: Affected
wsgiref.validate.InputWrapper: Affected
wsgiref.validate.PartialIteratorWrapper: Affected
wsgiref.validate.IteratorWrapper: Affected
xml.dom.pulldom.DOMEventStream: Affected
xml.etree.ElementTree.Element: Affected

If there's an equivalent of map_instance_to_supertype which handles structural super/sub classes already in the codebase somewhere that would be an easy fix.

Unfortunately, there is no such thing (commenting since I wanted something like this myself couple times), so I would suggest to aim for a revert. (Implementing such a helper is possible, but not super easy, so I don't want to do it in a hurry.)

Revert "Remove redundant inheritances from Iterator in builtins" (#18324

1.14.1 is out and has the typeshed revert #18324
This should be fixed in mypy, but we probably want to revert the changes in https://github.com/python/typeshed as well or we'll have problems next time typeshed gets synced.

but we probably want to revert the changes in https://github.com/python/typeshed as well or we'll have problems next time typeshed gets synced.

Not strictly necessary. #18324 included a git patch so it gets reverted automatically during the typeshed sync.

If there's an equivalent of map_instance_to_supertype which handles structural super/sub classes already in the codebase somewhere that would be an easy fix.

Unfortunately, there is no such thing (commenting since I wanted something like this myself couple times), so I would suggest to aim for a revert. (Implementing such a helper is possible, but not super easy, so I don't want to do it in a hurry.)

Can't we simply use the solver? I am using this in a patch I am working on, where I need to upcast a Type as Iterable[T].

def _solve_as_iterable(self, typ: Type, /) -> IterableType | AnyType:
    r"""Use the solver to cast a type as Iterable[T].

    Returns `AnyType` if solving fails.
    """
    from mypy.constraints import infer_constraints_for_callable
    from mypy.solve import solve_constraints

    # We first create an upcast function:
    #    def [T] (Iterable[T]) -> Iterable[T]: ...
    # and then solve for T, given the input type as the argument.
    T = TypeVarType(
        "T",
        "T",
        TypeVarId(-1),
        values=[],
        upper_bound=AnyType(TypeOfAny.from_omitted_generics),
        default=AnyType(TypeOfAny.from_omitted_generics),
    )
    target = self.make_iterable_instance_type(T)
    upcast_callable = CallableType(
        variables=[T],
        arg_types=[target],
        arg_kinds=[ARG_POS],
        arg_names=[None],
        ret_type=target,
        fallback=self.function_type,
    )
    constraints = infer_constraints_for_callable(
        upcast_callable, [typ], [ARG_POS], [None], [[0]], self
    )

    (sol,), _ = solve_constraints([T], constraints)

    if sol is None:  # solving failed, return AnyType fallback
        error_type = AnyType(TypeOfAny.from_error)
        return self.make_iterable_instance_type(error_type)
    return self.make_iterable_instance_type(sol)

Navigation Menu

Repository navigation

[1.14 regression] Unpacking an iterator converts the inner type to `Any` #18320

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[1.14 regression] Unpacking an iterator converts the inner type to Any #18320

Description

Activity

hauntsaninja commented on Dec 21, 2024

hauntsaninja commented on Dec 21, 2024

tungol commented on Dec 21, 2024

tungol commented on Dec 21, 2024

svalentin commented on Dec 21, 2024

tungol commented on Dec 21, 2024

tungol commented on Dec 21, 2024

ilevkivskyi commented on Dec 27, 2024

svalentin commented on Dec 30, 2024

cdce8p commented on Dec 30, 2024

randolf-scholz commented on Sep 16, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

[1.14 regression] Unpacking an iterator converts the inner type to `Any` #18320