Failed TypeVar substitution by asynccontextmanager

Bug Report

Return type of contextlib.asynccontextmanager does not substitute typing.TypeVar determined from parameters.

To Reproduce

Pass the following script to mypy, using mypy --config-file= main.py or otherwise.

#!/usr/bin/env python3

import contextlib, typing

_T = typing.TypeVar('_T')


@contextlib.asynccontextmanager
async def identity(element: _T) -> typing.AsyncIterator[_T]:
    """
    A context manager that does nothing
    and returns the given element.
    Type checking should assume the yielded value
    has the same type as the parameter.
    """
    yield element


async def main() -> None:
    """
    Here we give an ``int`` to ``identity``.
    So the yielded ``number`` should be ``int`` as well.
    """
    async with identity(1) as number:
        """Problem: We got ``_T`-1`` as its type instead of ``int``."""
        reveal_type(number)
        number = 2

Expected Behavior

The type of number should be int. In particular, assignment of the integer 2 to it should be compatible, and so mypy should return without errors.

Actual Behavior

The variable number is assigned, what I assume, is the TypeVar. This leads to an incompatible assignment of 2.

main.py:26: note: Revealed type is '_T`-1'
main.py:27: error: Incompatible types in assignment (expression has type "int", variable has type "_T")
Found 1 error in 1 file (checked 1 source file)

As a side note, a non-async counterpart of this behaves as expected, and does not produce any errors.

Your Environment

Mypy version used: 0.790
Mypy command-line flags: --config-file=
Mypy configuration options from mypy.ini (and other config files): Not applicable.
Python version used: Python 3.9.1
Operating system and version: Debian bullseye/sid and Windows 10 version 1909 build 18363.1256

This might be a typeshed bug; relevant definitions are for @asynccontextmanager (https://github.com/python/typeshed/blob/master/stdlib/2and3/contextlib.pyi#L32) and AsyncContextManager (https://github.com/python/typeshed/blob/master/stdlib/3/typing.pyi#L415). I don't see anything obviously wrong in those stubs though.

Note that mypy currently special cases contextlib.contextmanager (but doesn't special case contextlib.asynccontextmanager):

mypy/mypy/plugins/default.py

Line 172 in 92923b2

def contextmanager_callback(ctx: FunctionContext) -> Type:

This is something PEP 612 should help with.

@hauntsaninja

I think mypy is doing the right thing here given the current definition of asynccontextmanager in contextlib.pyi. As @hauntsaninja mentioned, it will need to be updated to make use of ParamSpec (defined in PEP 612).

I confirmed that it works as expected if the typeshed definition of asynccontextmanager is changed to:

_P = ParamSpec("_P")
def asynccontextmanager(func: Callable[_P, AsyncIterator[_T]]) -> Callable[_P, AsyncContextManager[_T]]: ...

We need to wait for python/typeshed#4827.

I looked into how contextmanger is handled, and I think its special casing can be applied to asynccontextmanager without change. Literally a two-line fix to bring it over. (I am unsure of the if-tree order for optimisation purposes though.) May be it can be a temporary fix until mypy has PEP612 compliance?

I tried to add a regression test, by copying over the non-async version, but it seems to fail, unable to find asynccontextmanager in contextlib. I am missing something obvious, I think. A little hint would be appreciated.

A unit test using minimal stubs works though. And I believe this test more accurately reflects the issue.

Please let me know what to do from here, if this fix can be used.

PEP 612 targets Python 3.10. Does that mean special casing for < 3.10 will still be necessary?

I would like to submit a temporary fix for it, but I am having trouble getting a unit test to work. Specifically, a test in test-data/unit/pythoneval.test. According to its comment, its tests use full stubs. But the test I add to it is not able to find contextlib.asynccontextmanager. Here is a summary of what I did, and it should run as is.

#!/usr/bin/bash

# Check out into current directory if not already done.
if [ -z "$(ls -A .)" ]; then
  git clone git@github.com:BoniLindsley/mypy.git .  #
fi

# Make sure things are up to date.
git submodule update --init
python3 -m pip install --upgrade --requirement test-requirements.txt

# Add test if not already done.
test_file=test-data/unit/pythoneval.test
if ! grep testAsyncContextManager $test_file > /dev/null; then
  cat << EOF >> test-data/unit/pythoneval.test

[case testAsyncContextManager]
# flags: --python-version 3.7
import contextlib
from contextlib import asynccontextmanager
from typing import AsyncIterator

@asynccontextmanager
async def f(x: int) -> AsyncIterator[str]:
    yield 'foo'

@contextlib.asynccontextmanager
async def g(*x: str) -> AsyncIterator[int]:
    yield 1

reveal_type(f)
reveal_type(g)

with f('') as s:
    reveal_type(s)
[out]
_program.py:13: note: Revealed type is 'def (x: builtins.int) -> typing.AsyncContextManager[builtins.str*]'
_program.py:14: note: Revealed type is 'def (*x: builtins.str) -> typing.AsyncContextManager[builtins.int*]'
_program.py:16: error: Argument 1 to "f" has incompatible type "str"; expected "int"
_program.py:17: note: Revealed type is 'builtins.str*'
EOF
fi

# The actual test.
python3 -m pytest --quiet -k testAsyncContextManager

The test is expected to fail, since this does not include the fix, and to produce a diff with wrong revealed types. But the error from it is instead

Actual:
  _testAsyncContextManager.py:3: error: Module 'contextlib' has no attribute 'asynccontextmanager'; maybe "contextmanager" or "AsyncContextManager"? (diff)
  _testAsyncContextManager.py:10: error: Module has no attribute "asynccontextmanager"; maybe "contextmanager" or "AsyncContextManager"? (diff)

So, the question is, how do I make the test find asynccontextmanager? May be I am missing something obvious.

Environment: Debian, pyenv, Python 3.9.1.

Extend contextmanager to asynccontextmanager (#11352)

Extend contextmanager to asynccontextmanager (python#11352)

Navigation Menu

Repository navigation

Failed `TypeVar` substitution by `asynccontextmanager` #9922

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Failed TypeVar substitution by asynccontextmanager #9922

Description

Activity

JelleZijlstra commented on Jan 18, 2021

hauntsaninja commented on Jan 18, 2021

erictraut commented on Jan 18, 2021

BoniLindsley commented on Jan 18, 2021

BoniLindsley commented on Feb 12, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Failed `TypeVar` substitution by `asynccontextmanager` #9922