Allow variable redefinition if lifetimes don't overlap

As a follow-up to #1174 and #6197, we should perhaps allow two assignments to create independent variables whenever the lifetimes of the variables don't overlap (when using --allow-redefinitions).

Example:

if foo():
    x = 0
    print(x)
else:
    x = 'a'  # Define a new variable?
    print(x)

Another example:

for x in [1, 2]:
    print(x)
if foo():
    for x in ['a', 'b']:  # Define a new variable?
        print(x)

Not sure if this would need to be split into a separate issue, or if it is relevant here, but I have some code like this:

try:
    ...
except AssertionError as e:
    errors.append(e)

if condition:
    e = AssertionError("message")
    errors.append(e)

Essentially the same as Django's ValidationError pattern. mypy warns for the assignment to variable e outside the except block, and also the reading of the deleted variable, despite the fact that I've just assigned it.

This is exactly what I see as a common pattern in code - quite often due to for loops or exceptions - exactly as in the above examples. So I'm also raising my hand for prioritizing this 😉.

My case, which as far as I can tell is related to this issue, is with the following.

I am (ab)using context managers in the following pattern, which is used in some configuration building code written by end-users of my tool.

Heavily simplified:

toolmodule.py:

import contextlib
import typing

class A:
   a: int
   b: str

def get_A(name: str) -> typing.Generator[A, None, None]:
...

class B:
   c: bool

def get_B(name: str) -> typing.Generator[B, None, None]:
...

userwritten.py:

import toolmodule

with get_A("") as x:
  x.a = 1
  x.b = ""

with get_B("") as x:
  x.c = True

for i in [1]:
  with get_A("") as x: # <- error: Incompatible types in assignment (expression has type "A", variable has type "B")
    x.a = 1

(In actual code, get_X return special context managers returning accessor classes which fail in runtime once the scope of with block is exited.)

I suspected that there could be some nuances related to context managers, but the following code has the same problem:

import typing as tp


x = 1
if tp.TYPE_CHECKING:
    reveal_type(x)
print(x)


x = [1]
if tp.TYPE_CHECKING:
    reveal_type(x)
print(x)


if True:
    x = ""  # <- error: Incompatible types in assignment (expression has type "str", variable has type "List[int]")
    if tp.TYPE_CHECKING:
        reveal_type(x)
    print(x)

for i in [1]:
    x = ""  # <- error: Incompatible types in assignment (expression has type "str", variable has type "List[int]")
    if tp.TYPE_CHECKING:
        reveal_type(x)
    print(x)

Rhetorically and off-topic: I really wish Python had some scoping statement, akin to:

scoped:
  x = 1

x  # <- error: undefined

scoped for i in []:
   ...

i  # <- error: undefined

scoped with f() as y:
   ...

y  # <- error: undefined

@ShadowLNC

Is there any progress on this? Having type "redefinition" errors, just because I used a variable that's named the same as the one I used in a for loop couple of lines earlier, is getting quite tiresome. The flag that allows redefinitions doesn't sound that great for my usage case either.

IMO, if someone is using a variable in a for loop, it should redefine the type of the variable, even without the "allow redefinitions" flag - consider this:

my_list: List[int]
my_other_list: List[Thing]

for x in my_list:
    reveal_type(x)  # int

reveal_type(x)  # int (still, persistent)

for x in my_other_list:  # redefines type of x
    reveal_type(x)  # Thing

x = "test"  # str (redefines type of x as well)

Perhaps the variable could be tagged as "reassignable" internally by MyPy somehow? This would also handle @ShadowLNC's case.

Can this be a related benign redefinition case too:

from pathlib import Path
for x in ['a/b']:
    reveal_type(x) # str, ok
    x = Path(x) # can’t redefine
    reveal_type(x) # str instead of Path
    reveal_type(x.stem) # error accessing, and Any because of that

With no for it works as expected, but in this manner it suddenly doesn’t (mypy 1.2, Python 3.11). I can’t see why a single iteration of a loop is different from an isolated block, as at the start of each iteration, x is reassigned a guaranteed str instance (if we know that what’s iterated over is Iterable[str])—but maybe there are pitfalls.

I'm not sure if this is the same issue, but I have a similar problem... I have the following code:

def get_menus(date: datetime.date) -> dict[str, Optional[dict[str, str]]]:
    # SnackMenu and LunchMenu are SQLAlchemy 2.0 models
    snack = Session.query(SnackMenu).filter(SnackMenu.date == date).first()
    lunch = Session.query(LunchMenu).filter(LunchMenu.date == date).first()

    reveal_type(snack)  # Revealed type is "Union[gimvicurnik.database.SnackMenu, None]"
    reveal_type(lunch)  # Revealed type is "Union[gimvicurnik.database.LunchMenu, None]"

    if snack:
        snack = {
            "normal": snack.normal,
            "poultry": snack.poultry,
            "vegetarian": snack.vegetarian,
            "fruitvegetable": snack.fruitvegetable,
        }

    if lunch:
        lunch = {
            "normal": lunch.normal,
            "vegetarian": lunch.vegetarian,
        }

    return {
        "snack": snack,
        "lunch": lunch,
    }

When running mypy with the following configuration (notice that allow_redefinition is enabled):

[tool.mypy]
python_version = 3.9
show_column_numbers = true
show_error_codes = true
allow_redefinition = true
disallow_untyped_decorators = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
no_implicit_optional = true
warn_redundant_casts = true
warn_unused_configs = true
warn_unused_ignores = true
warn_no_return = true
warn_return_any = true
warn_unreachable = true

I get the following type errors:

gimvicurnik\blueprints\menus.py:29:25: error: Incompatible types in assignment (expression has type "Dict[str, Optional[str]]", variable has type "Optional[SnackMenu]")  [assignment]
gimvicurnik\blueprints\menus.py:37:25: error: Incompatible types in assignment (expression has type "Dict[str, Optional[str]]", variable has type "Optional[LunchMenu]")  [assignment]
gimvicurnik\blueprints\menus.py:43:17: error: Dict entry 0 has incompatible type "str": "Optional[SnackMenu]"; expected "str": "Optional[Dict[str, str]]"  [dict-item]
gimvicurnik\blueprints\menus.py:44:17: error: Dict entry 1 has incompatible type "str": "Optional[LunchMenu]"; expected "str": "Optional[Dict[str, str]]"  [dict-item]

I would expect that mypy would not reject this code, especially since allow_redefinition is enabled.

I'm not sure if this is the same issue

It looks like a different issue, of SnackMenu and LunchMenu not being compatible with a dict[str, str] type. This entire issue focuses on a situation where one creates a variable, uses it, and then gets an "incompatible type" error later on, simply because they've decided to use a variable that's named the same as the one that was used before. For specific examples, see the first message of this issue: #6232 (comment)

One workaround: in several cases, adding x: object before any uses of the variable x will make mypy behave the way you want it to

I'm working on a prototype implementation of this.

#18568 looks like a promising implementation approach.

Related to this topic, is allowing the type of redefinition of function arguments covered under this issue, or a separate one? For example:

def func(arg: string):
  arg = int(string)
  ...

is allowing the type of redefinition of function arguments covered under this issue, or a separate one?

That's a separate one. The current plan is to allow redefinition when there is no explicit type annotation. Allowing it even when there is an explicit type annotation needs to be discussed separately. Here are arguments for and against:

For: It works at runtime, so not allowing it arguably leads to false positives.
For: It makes it easier to migrate existing unannotated code to static type checking, since otherwise some variables may have to be renamed, which takes effort and risks introducing bugs.
Against: It's confusing if the annotation doesn't match the runtime type.
Against: The type annotation may have been added to make sure that only values of a specific type are assigned to the variable.

There's also the open issue of whether multiple type annotations are allowed for a single name in a scope. For example, should this be allowed:

def f(b: bool) -> None:
    if b:
        x: list[str] = []
        foo(x)
    else:
        x: set[int] = set()
        bar(x)

And what if the parameter was also named x?

def f(x: bool) -> None:
    if x:
        x: list[str] = []
        foo(x)
    else:
        x: set[int] = set()
        bar(x)

Had an idea regarding semantics of this: reassignment with a different type can often be thought of as name shadowing: having just a fresh variable with the same name but another type. As this is usually both useful in one cases and induces hard-to-catch bugs in others, a meta-annotation akin to Final[actual_type] (say Re[...], Fresh[...] or New[...]) could explicitly state the author’s intent in a case they imply shadowing. I don’t know if type annotations on variables have sequencing information on them or not but if they do, re-annotations can probably work and make some code allowed while allowing to ban the code that’s probably was a result of a typo or refactoring gone bad.

Then for example my code above would be formatted as:

for x in ['a/b']:
    # x: str
    x: Fresh = Path(x)  # with arg defaulting working similar to `Final`

to state that this redefinition is “actually” an intended shadowing to make less lexical noise. (Which it indeed is, and I think the way I personally use this idiom should 98% be shadowing.)

Add flag to allow more flexible variable redefinition (python#18727)

Navigation Menu

Repository navigation

Allow variable redefinition if lifetimes don't overlap #6232

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Allow variable redefinition if lifetimes don't overlap #6232

Description

Activity

ShadowLNC commented on Dec 30, 2019

dumblob commented on Jan 7, 2020

kshpytsya commented on May 25, 2020

DevilXD commented on Feb 24, 2021

arseniiv commented on Apr 16, 2023

filips123 commented on Apr 22, 2023

DevilXD commented on Apr 22, 2023

hauntsaninja commented on Dec 27, 2024

JukkaL commented on Jan 23, 2025

JukkaL commented on Jan 30, 2025

strokirk commented on Feb 11, 2025

JukkaL commented on Feb 11, 2025

arseniiv commented on Feb 11, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions