Skip to content

Ensures overloads are ordered from narrow to broad - #2138

Merged
JelleZijlstra merged 4 commits into
python:masterfrom
Michael0x2a:rearrange-overloads
May 17, 2018
Merged

Ensures overloads are ordered from narrow to broad#2138
JelleZijlstra merged 4 commits into
python:masterfrom
Michael0x2a:rearrange-overloads

Conversation

@Michael0x2a

@Michael0x2a Michael0x2a commented May 16, 2018

Copy link
Copy Markdown
Contributor

This commit reorders any overloads where the first overload was "shadowing" the second, preventing it from ever being matched by type checkers that work by selecting the first matching overload alternative.

For example, the first overload alternative below is strictly broader then the second, preventing it from ever being selected:

class Parent: pass
class Child(Parent): pass

@overload
def foo(x: *int) -> Parent: ...
@overload
def foo(x: int, y: int) -> Child: ...

The correct thing to do is to either delete the second overload or rearrange them to look like this:

@overload
def foo(x: int, y: int) -> Child: ...
@overload
def foo(x: *int) -> Parent: ...

Rationale: I'm currently working on a proposal that would amend PEP 484 to (a) mandate type checkers check overloads in order and (b) prohibit overloads where an earlier alternative completely shadows a later one.

This would prohibit overloads that look like the example below, where the first alternative completely shadows the second.

I figured it would be a good idea to make these changes ahead of time: if my proposal is accepted, it'd make the transition smoother. If not, this is hopefully a relatively harmless change.

Note: I think some of these overloads could be simplified (e.g. reversed(...)), but I mostly stuck with rearranging them in case I was wrong. The only overload I actually changed was hmac.compare_digest -- I believe the Python 2 version actually accepts unicode.

@Michael0x2a
Ensures commits are ordered from narrow to broad
4f5ddcb 
This commit reorders any overloads where the first overload was
"shadowing" the second, preventing it from ever being matched by type
checkers that work by selecting the first matching overload alternative.

For example, the first overload alternative below is strictly broader
then the second, preventing it from ever being selected:

    class Parent: pass
    class Child(Parent): pass

    @overload
    def foo(x: *int) -> Parent: ...
    @overload
    def foo(x: int, y: int) -> Child: ...

The correct thing to do is to either delete the second overload or
rearrange them to look like this:

    @overload
    def foo(x: int, y: int) -> Child: ...
    @overload
    def foo(x: *int) -> Parent: ...

Rationale: I'm currently [working on a proposal][0] that would amend
PEP 484 to (a) mandate type checkers check overloads in order and
(b) prohibit overloads where an earlier alternative completely shadows
a later one.

  [0]: python/typing#253 (comment)

This would prohibit overloads that look like the example below, where
the first alternative completely shadows the second.

I figured it would be a good idea to make these changes ahead of time:
if my proposal is accepted, it'd make the transition smoother. If not,
this is hopefully a relatively harmless change.

Note: I think some of these overloads could be simplified (e.g.
`reversed(...)`), but I mostly stuck with rearranging them in case I was
wrong. The only overload I actually changed was `hmac.compare_digest` --
I believe the Python 2 version actually accepts unicode.
@JelleZijlstra

JelleZijlstra commented May 16, 2018

Copy link
Copy Markdown
Member

I assume you mean "overloads", not "commits" :)

JelleZijlstra
JelleZijlstra reviewed May 16, 2018
View reviewed changes
Comment thread stdlib/2/os/__init__.pyi Outdated
@overload
def stat_float_times() -> bool: ...
@overload
def stat_float_times(newvalue: bool = ...) -> None: ...

@JelleZijlstra JelleZijlstra May 16, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we just remove the = ... here?

@Michael0x2a Michael0x2a May 16, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@Michael0x2a Michael0x2a changed the title Ensures commits are ordered from narrow to broad Ensures overloads are ordered from narrow to broad May 16, 2018
@Michael0x2a

Michael0x2a commented May 16, 2018

Copy link
Copy Markdown
Contributor Author

lol, good point -- fixed the title

Michael0x2a added a commit to Michael0x2a/mypy that referenced this pull request May 16, 2018
@Michael0x2a
Relax overload checks to allow return types to be regular subtypes
5b723e6 
This pull request relaxes the overlapping overload checks so that if the
parameter types of an alternative are a proper subtype of another, the
return type needs to only be a regular subtype, not a proper subtype.

The net effect is that the return type of the first alternative is
allowed to be 'Any'.

This *does* make overloads slightly less safe, but 'Any' was always
meant to be an escape hatch, so I'm figuring this is fine.

**Rationale**: My [proposed changes on overhauling overloads][0] require
a few changes to typeshed -- [relevant pull request here][1]. Rather then
trying to change both at the same time, I want to get typeshed working
first.

This was pretty easy to do, apart from the attrs stubs. The issue
basically boils down to this:

    # Alternative 1
    @overload
    def attrib(x: Optional[T]) -> T: ...

    # Alternative 2
    @overload
    def attrib(x: None = None) -> Any: ...

This code typechecks under the current system because alternative 1
completely shadows alternative 2. It fails to typecheck under the new
system for the exact same reason.

If we swap the two alternatives, it fails under the current system
because 'Any' is not a proper subtype of 'T'.

It *is*, however, regular subtype of 'T' -- hence this change.

  [0]: python/typing#253 (comment)

  [1]: python/typeshed#2138
@Michael0x2a Michael0x2a mentioned this pull request May 16, 2018
Merged
@Michael0x2a

Michael0x2a commented May 16, 2018 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

I reverted the changes to the attrs stubs for now -- it looks like the attrs stubs are currently impossible to type in a way that's compatible with both the current and proposed overload semantics.

I submitted a pull request to slightly relax mypy's current overload semantics so that this change can go through. Once my other pull request is accepted, I'll unrevert my revert (unless you decide to accept this pull request as-is -- if so, I can just make another pull request for attrs).

ilevkivskyi pushed a commit to python/mypy that referenced this pull request May 16, 2018
@Michael0x2a @ilevkivskyi
Relax overload checks to allow return types to be regular subtypes (#…
74371df 
…5064)

This pull request relaxes the overlapping overload checks so that if the
parameter types of an alternative are a proper subtype of another, the
return type needs to only be a regular subtype, not a proper subtype.

The net effect is that the return type of the first alternative is
allowed to be 'Any'.

This *does* make overloads slightly less safe, but 'Any' was always
meant to be an escape hatch, so I'm figuring this is fine.

**Rationale**: My [proposed changes on overhauling overloads][0] require
a few changes to typeshed -- [relevant pull request here][1]. Rather then
trying to change both at the same time, I want to get typeshed working
first.

This was pretty easy to do, apart from the attrs stubs. The issue
basically boils down to this:

    # Alternative 1
    @overload
    def attrib(x: Optional[T]) -> T: ...

    # Alternative 2
    @overload
    def attrib(x: None = None) -> Any: ...

This code typechecks under the current system because alternative 1
completely shadows alternative 2. It fails to typecheck under the new
system for the exact same reason.

If we swap the two alternatives, it fails under the current system
because 'Any' is not a proper subtype of 'T'.

It *is*, however, regular subtype of 'T' -- hence this change.

  [0]: python/typing#253 (comment)

  [1]: python/typeshed#2138
@Michael0x2a

Michael0x2a commented May 17, 2018

Copy link
Copy Markdown
Contributor Author

@JelleZijlstra -- Ok, looks like third time was the charm. I think this is ready for a second look whenever you have time.

@JelleZijlstra
JelleZijlstra merged commit 97d9f2e into python:master May 17, 2018
@Michael0x2a
Michael0x2a deleted the rearrange-overloads branch May 17, 2018 13:47
Michael0x2a added a commit to Michael0x2a/mypy that referenced this pull request May 17, 2018
@Michael0x2a
Sync typeshed
41f814f 
Syncs typeshed, mostly to pull in some [tweaks I made][0] that makes
typeshed compatible with both the current and [proposed][1] overload
semantics.

  [0]: python/typeshed#2138
  [1]: python/typing#253 (comment)
@Michael0x2a Michael0x2a mentioned this pull request May 17, 2018
Merged
ilevkivskyi pushed a commit to python/mypy that referenced this pull request May 17, 2018
@Michael0x2a @ilevkivskyi
Sync typeshed (#5071)
eaa16f3 
Syncs typeshed, mostly to pull in some [tweaks I made][0] that makes
typeshed compatible with both the current and [proposed][1] overload
semantics.

  [0]: python/typeshed#2138
  [1]: python/typing#253 (comment)
Michael0x2a added a commit to Michael0x2a/typeshed that referenced this pull request May 18, 2018
@Michael0x2a
Rearrange overloads to account for optional arguments
beaff80 
Basically, the same thing as [my previous pull request][0], except the
fixes are now focusing on functions with overlapping argument counts.

  [0]: python#2138
@Michael0x2a Michael0x2a mentioned this pull request May 18, 2018
Merged
JelleZijlstra pushed a commit that referenced this pull request May 19, 2018
@Michael0x2a @JelleZijlstra
Rearrange overloads to account for optional arguments (#2150)
709b193 
Basically, the same thing as [my previous pull request][0], except the
fixes are now focusing on functions with overlapping argument counts.

  [0]: #2138
gwk pushed a commit to gwk/typeshed that referenced this pull request May 29, 2018
@Michael0x2a @gwk
Ensures overloads are ordered from narrow to broad (python#2138)
a41030f 
This commit reorders any overloads where the first overload was
"shadowing" the second, preventing it from ever being matched by type
checkers that work by selecting the first matching overload alternative.

For example, the first overload alternative below is strictly broader
then the second, preventing it from ever being selected:

    class Parent: pass
    class Child(Parent): pass

    @overload
    def foo(x: *int) -> Parent: ...
    @overload
    def foo(x: int, y: int) -> Child: ...

The correct thing to do is to either delete the second overload or
rearrange them to look like this:

    @overload
    def foo(x: int, y: int) -> Child: ...
    @overload
    def foo(x: *int) -> Parent: ...

Rationale: I'm currently [working on a proposal][0] that would amend
PEP 484 to (a) mandate type checkers check overloads in order and
(b) prohibit overloads where an earlier alternative completely shadows
a later one.

  [0]: python/typing#253 (comment)

This would prohibit overloads that look like the example below, where
the first alternative completely shadows the second.

I figured it would be a good idea to make these changes ahead of time:
if my proposal is accepted, it'd make the transition smoother. If not,
this is hopefully a relatively harmless change.

Note: I think some of these overloads could be simplified (e.g.
`reversed(...)`), but I mostly stuck with rearranging them in case I was
wrong. The only overload I actually changed was `hmac.compare_digest` --
I believe the Python 2 version actually accepts unicode.
gwk pushed a commit to gwk/typeshed that referenced this pull request May 29, 2018
@Michael0x2a @gwk
Rearrange overloads to account for optional arguments (python#2150)
a5bf0b1 
Basically, the same thing as [my previous pull request][0], except the
fixes are now focusing on functions with overlapping argument counts.

  [0]: python#2138
yedpodtrzitko pushed a commit to yedpodtrzitko/typeshed that referenced this pull request Jan 23, 2019
@Michael0x2a Jiri Suchan
Ensures overloads are ordered from narrow to broad (python#2138)
006397f 
This commit reorders any overloads where the first overload was
"shadowing" the second, preventing it from ever being matched by type
checkers that work by selecting the first matching overload alternative.

For example, the first overload alternative below is strictly broader
then the second, preventing it from ever being selected:

    class Parent: pass
    class Child(Parent): pass

    @overload
    def foo(x: *int) -> Parent: ...
    @overload
    def foo(x: int, y: int) -> Child: ...

The correct thing to do is to either delete the second overload or
rearrange them to look like this:

    @overload
    def foo(x: int, y: int) -> Child: ...
    @overload
    def foo(x: *int) -> Parent: ...

Rationale: I'm currently [working on a proposal][0] that would amend
PEP 484 to (a) mandate type checkers check overloads in order and
(b) prohibit overloads where an earlier alternative completely shadows
a later one.

  [0]: python/typing#253 (comment)

This would prohibit overloads that look like the example below, where
the first alternative completely shadows the second.

I figured it would be a good idea to make these changes ahead of time:
if my proposal is accepted, it'd make the transition smoother. If not,
this is hopefully a relatively harmless change.

Note: I think some of these overloads could be simplified (e.g.
`reversed(...)`), but I mostly stuck with rearranging them in case I was
wrong. The only overload I actually changed was `hmac.compare_digest` --
I believe the Python 2 version actually accepts unicode.
yedpodtrzitko pushed a commit to yedpodtrzitko/typeshed that referenced this pull request Jan 23, 2019
@Michael0x2a Jiri Suchan
Rearrange overloads to account for optional arguments (python#2150)
429b946 
Basically, the same thing as [my previous pull request][0], except the
fixes are now focusing on functions with overlapping argument counts.

  [0]: python#2138
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JelleZijlstra JelleZijlstra JelleZijlstra left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Michael0x2a @JelleZijlstra