Assignment of Any does not get rid of Optional (add AnyUnion type?)

Consider this code (run with --strict-optional):

from typing import Optional
def f():
  # untyped function; often from silent import
  return 0  # always returns an int

def g(x: Optional[int]) -> int
  if x is None:
    x = f()
  return x  # E: Incompatible return value type (got "Optional[int]", expected "int")

Mypy doesn't infer that x is not None in g. I think this a bug -- mypy
should be permissive here.

If we wanted to fix this, we'd run into a problem: what type should x have
after x = f()? We don't want it to be Any, because we don't want that
behavior in the general case. It shouldn't be int -- f() could be intended
to unconditionally return None instead. To fix this, I think we need to
introduce a new special type, which I will call AnyUnion. AnyUnion[A, B, C] would represent a type which is allowed to be used as any of A, B, or
C (similarly to how Any is allowed to be used as any type). In this case,
we could have x's type bound to AnyUnion[int, None], which would fix this
error.

Other helpful uses of AnyUnion:

pow will no longer need to return Any in builtins. Instead, it could
return the much less broad AnyUnion[int, float].
AnyUnion[X, None] would provide a per-variable opt-out of strict None
checking. This could be helpful for classes with delayed initialization where
a lot of effort would be needed to refactor them into a state that would work
properly with strict None checking.

Cons:

Additional complexity of the type system affects implementation complexity and
makes things slightly harder for users to understand.
The motivating problem also exists with subclasses (with the way the binder
currently works), but AnyUnion doesn't help much with that.

pow will no longer need to return Any in builtins. Instead, it could
return the much less broad AnyUnion[int, float].

But doesn't #3501 (or similar) provides a better solution?

AnyUnion[X, None] would provide a per-variable opt-out of strict None
checking. This could be helpful for classes with delayed initialization where
a lot of effort would be needed to refactor them into a state that would work
properly with strict None checking.

Do I understand correctly that this has been solved by PEP 526?

In general, I don't think that we really need this feature. What worries me is that in addition to making type system more complex it also can break transitivity of subtyping (or maybe better call this compatibility). IIUC your proposal, then int <: AnyUnion[int, str] <: str. Currently, we have only one special type that does this -- Any. Having many (user defined) types that break transitivity might lead to unexpected consequences.

Do I understand correctly that this has been solved by PEP 526?

Probably it was only partially fixed by PEP 526. But for remaining cases one can just do something like:

UNINITIALIZED: Any = None

class C:
    def __init__(self) -> None:
        self.complicated: int = UNINITIALIZED
        # existing code below
        while self.complicated is None:
            ...

I'm not sure this is needed all that often, but if we do, we could just spell it Any -- Any[A, B, C] is acceptable as a A, B or C, while plain Any is unchanged. Alternatively, maybe this type is really the same as Intersection (python/typing#213)?

Regardless I'm not sure that in your original motivating example we should do anything different. If you want to signal at the call site that f() returns an int, you can write

def g(x: Optional[int]) -> int
  if x is None:
    y = f()  # type: int
    x = y
  return x  # OK

This is probably the same feature as 'unsafe unions' that have been discussed before (though I can't find the discussion -- maybe it was in person or over email). Unsafe unions are different from Intersection. For example, unlike intersections, compatibility goes both ways: int is compatible with AnyUnion[int, str] and vice versa.

My take is that this feature would complicate mypy significantly. Normal union types are already a very complicated feature, and they still aren't even fully implemented. It would be a likely cause of some user confusion, since we'd have two pretty similar but different concepts -- union types and these any/unsafe unions. If we decide to add intersection types at some point, there would be even more potential for confusion, since they are also similar in some ways to any unions. All in all, I don't think that there is enough evidence of the usefulness of the feature to support implementing it.

Solving the original issue is a priority because this may come up often when using ignored imports. I'm not convinced that adding AnyUnion is the right course of action, though.

Concerning the original example:

from typing import Optional
def f():
  # untyped function; often from silent import
  return 0  # always returns an int

def g(x: Optional[int]) -> int
  if x is None:
    x = f()
  return x

what is wrong with x having type Union[Any, int] after the if block? Thus seems logical, initially it was Union[None, int] but then the None is replaced by Any in the if block by x = f(). Fortunately, such unions are not simplified anymore:

x: Union[int, Any]
reveal_type(x) # Revealed type is 'Union[builtins.int, Any]'
x.whatever # Error: Item "int" of "Union[int, Any]" has no attribute "whatever"

Also, Union[int, Any] is a (non-proper) subtype of int so that there should be no error in the original example.

@ilevkivskyi

@ilevkivskyi: That doesn't work, as variables don't change type in general. In particular, we want to maintain this behavior:

x = 0  # type: int
x = any()
reveal_type(x)  # int

@ddfisher

@ddfisher

as variables don't change type in general

But there is the type binder that can already do this:

x: Optional[int]

if x is None:
    x = int()

reveal_type(x)  # Revealed type is 'builtins.int'

Why can't it be special-cased to re-bind to Any inside unions? (I agree that binding to Any in general is clearly bad.)

Unless there's a really good reason, Unions should work as similarly as possible to non-Unions. I don't think this is a strong enough reason.

The binder only binds to subtypes of the declared type. If we change that, the declared type starts to become pretty meaningless.

Nevertheless the bar for introducing a new concept like AnyUnion is way higher.

We should check how many more Anys the simple proposal would introduce into real world codebases before merging it.

And yes, this would conflict with proposal 3 in #2008 -- it would need tweaks.

The option number 3 already mentions

Unclear interactions with Any types.

as a downside. Maybe I am missing something, but this could be a minor tweak. Something like: types are always bound on initial assignments unless r.h.s. is Any.

It's not that simple. Take this circumstance, for example:

x = [1, 2, 3]  # type: Sequence[Any]

We don't want to bind List[int] for x, because we want to preserve Anys (as otherwise Anys in generics would be useless). We don't want to leave it as Sequence[Any], because that would make it behave inconsistently from all other list assignments. We'd have to bind List[Any] as the type for x. This isn't necessarily terrible, but is at a minimum a little tricky.

We'd have to bind List[Any] as the type for x. This isn't necessarily terrible, but is at a minimum a little tricky.

I think bind List[Any] in this case is a reasonable thing to do. Indeed, general interactions with Any may be complex. However, I would still propose to go with Jukka's simple solution (if it doesn't explode the number of Anys) and figure out the detailed rules about Any later, when we get to #2008.

#3431 is also similar.

This seems to happen frequently enough that I'm bumping priority to "high". Also removing the "needs discussion" label -- let's special case None for now, as discussed above.

I'm not 100% sure what the proposal is. From "discussed above" link it seems that we should make this work?

def f():...
def g(x: Optional[int]) -> int:
    if x is None:
        x = f()
        reveal_type(x)   # Union[int, Any]  # <---- this is new (currently Optional[int])
    reveal_type(x)  # Union[int, Any]  # Currently also Optional[int]
    return x  # Ok  # Currently an error (Incompatible return value type ...)

But how? And should the fix take into account that f() is unannotated, or should it work for anything of type Any? Let's assume the latter. At the point x = f() we do something special if certain conditions are met. What are those conditions? I'm guessing all of the following:

the target's type is known to be None due to a None check recorded in the binder overriding some Optional[t]
the value's type is Any

The action is then apparently to set the target's type to Union[t, Any]. The rest will follow.

Have I got that?

@gvanrossum

@gvanrossum

But how?

I think not much have to be done, the binder would do the things, but it is explicitly prohibited to do many things with AnyType, for example in ConditionalTypeBinder.assign_type:

# Assigning an Any value doesn't affect the type to avoid false negatives

I think we can just add an exception there, so that when declared_type is a union containing None and enclosing_type is None, we still put the type even if it is an AnyType.

Get rid of Optional when assigning Any in an `is None` branch

Get rid of Optional when assigning Any in an `is None` branch (#5629)

Navigation Menu

Repository navigation

Assignment of Any does not get rid of Optional (add `AnyUnion` type?) #3526

14 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Assignment of Any does not get rid of Optional (add AnyUnion type?) #3526

Description

Activity

ilevkivskyi commented on Jun 13, 2017

ilevkivskyi commented on Jun 13, 2017

gvanrossum commented on Jun 13, 2017

JukkaL commented on Jun 13, 2017

JukkaL commented on Jun 23, 2017

ilevkivskyi commented on Jun 23, 2017

ddfisher commented on Jun 23, 2017

ilevkivskyi commented on Jun 23, 2017

ddfisher commented on Jun 23, 2017

gvanrossum commented on Jun 23, 2017

14 remaining items

ddfisher commented on Jul 17, 2017

ddfisher commented on Jul 17, 2017

ilevkivskyi commented on Jul 17, 2017

JukkaL commented on Sep 18, 2017

JukkaL commented on Dec 5, 2017

gvanrossum commented on Aug 1, 2018

ilevkivskyi commented on Aug 2, 2018

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Assignment of Any does not get rid of Optional (add `AnyUnion` type?) #3526