Repository navigation
Rewrite operator Library with More Specific Annotations - #15058
Open
George-Ogden wants to merge 6 commits into
Open
George-Ogden wants to merge 6 commits into
George-Ogden wants to merge 6 commits into
Conversation
George-Ogden
marked this pull request as ready for review
November 20, 2025 15:59
George-Ogden
marked this pull request as draft
November 20, 2025 15:59
George-Ogden
force-pushed
the
operator-overhaul
branch
2 times, most recently
from
November 20, 2025 16:47
1b27d61 to
2a09a2b
Compare
Collaborator
|
This seems useful. In fact, I don't know why we're using |
George-Ogden
force-pushed
the
operator-overhaul
branch
from
February 25, 2026 10:08
0f26829 to
c6ca360
Compare
George-Ogden
force-pushed
the
operator-overhaul
branch
from
February 25, 2026 16:18
8535881 to
98b844d
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This reverts commit f6b5f2f.
Contributor
|
Diff from mypy_primer, showing the effect of this PR on open source code: spark (https://github.com/apache/spark)
+ python/pyspark/core/rdd.py:2168: error: Argument 2 to "fold" of "RDD" has incompatible type overloaded function; expected "Callable[[float | Literal[0], float | Literal[0]], float | Literal[0]]" [arg-type]
+ python/pyspark/core/rdd.py:2168: error: Argument 2 to "fold" of "RDD" has incompatible type overloaded function; expected "Callable[[complex | Literal[0], complex | Literal[0]], complex | Literal[0]]" [arg-type]
+ python/pyspark/core/rdd.py:2168: error: Argument 2 to "fold" of "RDD" has incompatible type overloaded function; expected "Callable[[signedinteger[_32Bit] | Literal[0], signedinteger[_32Bit] | Literal[0]], signedinteger[_32Bit] | Literal[0]]" [arg-type]
+ python/pyspark/core/rdd.py:2168: error: Argument 2 to "fold" of "RDD" has incompatible type overloaded function; expected "Callable[[signedinteger[_64Bit] | Literal[0], signedinteger[_64Bit] | Literal[0]], signedinteger[_64Bit] | Literal[0]]" [arg-type]
+ python/pyspark/core/rdd.py:2168: error: Argument 2 to "fold" of "RDD" has incompatible type overloaded function; expected "Callable[[floating[_32Bit] | Literal[0], floating[_32Bit] | Literal[0]], floating[_32Bit] | Literal[0]]" [arg-type]
+ python/pyspark/core/rdd.py:2168: error: Argument 2 to "fold" of "RDD" has incompatible type overloaded function; expected "Callable[[float64 | Literal[0], float64 | Literal[0]], float64 | Literal[0]]" [arg-type]
+ python/pyspark/core/rdd.py:2168: error: Argument 2 to "fold" of "RDD" has incompatible type overloaded function; expected "Callable[[ndarray[tuple[Any, ...], dtype[Any]] | Literal[0], ndarray[tuple[Any, ...], dtype[Any]] | Literal[0]], ndarray[tuple[Any, ...], dtype[Any]] | Literal[0]]" [arg-type]
colour (https://github.com/colour-science/colour)
+ colour/utilities/array.py:547: error: Cannot call function of unknown type [operator]
+ colour/io/luts/lut.py:638: error: Cannot call function of unknown type [operator]
+ colour/io/luts/lut.py:642: error: Cannot call function of unknown type [operator]
+ colour/continuous/signal.py:1089: error: Cannot call function of unknown type [operator]
+ colour/continuous/signal.py:1093: error: Cannot call function of unknown type [operator]
+ colour/continuous/signal.py:1097: error: Cannot call function of unknown type [operator]
rotki (https://github.com/rotki/rotki)
+ rotkehlchen/chain/ethereum/oracles/uniswap.py:326: error: Argument 1 to "reduce" has incompatible type overloaded function; expected "Callable[[int, FVal], int]" [arg-type]
sphinx (https://github.com/sphinx-doc/sphinx)
+ sphinx/ext/autodoc/_legacy_class_based/_documenters.py: note: In member "should_suppress_value_header" of class "DataDocumenter":
+ sphinx/ext/autodoc/_legacy_class_based/_documenters.py:2078:44: error: Argument 1 to "reduce" has incompatible type overloaded function; expected "Callable[[list[Never], list[str]], list[Never]]" [arg-type]
+ sphinx/ext/autodoc/_legacy_class_based/_documenters.py:2078:44: error: Argument 1 to "reduce" has incompatible type overloaded function; expected "Callable[[Iterable[str], list[str]], Iterable[str]]" [arg-type]
+ sphinx/ext/autodoc/_legacy_class_based/_documenters.py: note: In member "should_suppress_value_header" of class "AttributeDocumenter":
+ sphinx/ext/autodoc/_legacy_class_based/_documenters.py:2730:48: error: Argument 1 to "reduce" has incompatible type overloaded function; expected "Callable[[list[Never], list[str]], list[Never]]" [arg-type]
+ sphinx/ext/autodoc/_legacy_class_based/_documenters.py:2730:48: error: Argument 1 to "reduce" has incompatible type overloaded function; expected "Callable[[Iterable[str], list[str]], Iterable[str]]" [arg-type]
xarray (https://github.com/pydata/xarray)
+ xarray/core/accessor_str.py: note: In member "get_dummies" of class "StringAccessor":
+ xarray/core/accessor_str.py:2836: error: Argument 1 to "reduce" has incompatible type overloaded function; expected "Callable[[Iterable[Any], Iterable[Any]], Iterable[Any]]" [arg-type]
sympy (https://github.com/sympy/sympy)
+ sympy/polys/matrices/dense.py:104: error: Incompatible types in assignment (expression has type "RingElement", target has type "R") [assignment]
pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/computation/ops.py:323: error: Need type annotation for "_binary_ops_dict" (hint: "_binary_ops_dict: dict[<type>, <type>] = ...") [var-annotated]
+ pandas/core/algorithms.py:1346: error: Incompatible types in assignment (expression has type overloaded function, variable has type overloaded function) [assignment]
+ pandas/core/indexes/datetimes.py:1255: error: Incompatible types in assignment (expression has type overloaded function, variable has type overloaded function) [assignment]
ibis (https://github.com/ibis-project/ibis)
+ ibis/util.py:268: error: Incompatible types in assignment (expression has type "overloaded function | overloaded function", variable has type overloaded function) [assignment]
+ ibis/expr/api.py:714: error: Argument 1 to "reduce" has incompatible type overloaded function; expected "Callable[[BooleanValue | bool, BooleanValue | bool], BooleanValue | bool]" [arg-type]
+ ibis/expr/api.py:734: error: Argument 1 to "reduce" has incompatible type overloaded function; expected "Callable[[BooleanValue | bool, BooleanValue | bool], BooleanValue | bool]" [arg-type]
+ ibis/backends/sql/compilers/base.py:633: error: Argument 1 to "reduce" has incompatible type overloaded function; expected "Callable[[Replace, Replace], Replace]" [arg-type]
schemathesis (https://github.com/schemathesis/schemathesis)
+ src/schemathesis/cli/commands/run/validation.py: note: In function "reduce_list":
+ src/schemathesis/cli/commands/run/validation.py:203: error: Argument 1 to "reduce" has incompatible type overloaded function; expected "Callable[[list[Never], list[str]], list[Never]]" [arg-type]
|
Author
|
The main limitation of this approach is mypy's constraint-solving capabilities. ## Example 1
class BooleanValueBase:
def __and__(self, other: BooleanValueBase) -> BooleanValueSubclass1:
raise NotImplementedError()
class BooleanValueSubclass1(BooleanValueBase): ...
class BooleanValueSubclass2(BooleanValueBase): ...
def subclass_and(predicates: Sequence[BooleanValueSubclass2]) -> None:
reduce(operator.and_, predicates) # fails as it cannot deduce that predicates: Sequence[BooleanValueBase]
reduce(operator.and_, cast(Sequence[BooleanValueBase, ...], predicates)) # fixed with cast
## Example 2
class SpecialInt:
def __add__(self, other: SpecialInt | int) -> SpecialInt:
raise NotImplementedError()
def union_add(predicates: Sequence[SpecialInt]) -> None:
reduce(operator.add, predicates, 0) # fails as it cannot deduce that 0: SpecialInt | int
reduce(operator.add, predicates, cast(SpecialInt | int, 0)) # fixed with cast
## Example 3
def unresolved_generic(docs: Sequence[list[str]]) -> None:
reduce(operator.iadd, docs, []) # fails: [] has type list[Never]
reduce(operator.iadd, docs, cast(list[str], [])) # fixed with cast |
Author
|
I tried adding a fallback to |
George-Ogden
marked this pull request as ready for review
February 25, 2026 17:55
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
More detailed
operatorannotations could be very useful (see #6448).For example,
This used to type check with either return type, but should only work with
list[float]now.Otherwise, the
operatorlibrary is mostly type usingbinop(a: Any, b: Any, /) -> Any.