diff --git a/docs/SPEC.md b/docs/SPEC.md index 3f09c746..ab7652e5 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -688,7 +688,8 @@ All in `PM/commonTest`, so they run wherever the interpreter loads. | Array access in bounds, or a runtime bounds check kept | the check stays (never an unchecked access) | | Array parameters only indexed, stored, `len`-ed; no aliasing (entry guard) | not compiled; diagnostic | | Integer ops proven safe by the verifier's intervals (constants, `len`, `range` variables, guards such as `if x < e` / `while lo < hi`, joined local assignments, `//` and `%` by constants) | emitted unchecked (a shift/mask for `//`/`%` by a power of two of a non-negative value); the sanitizer CI run builds with `TP_CHECK_PROVEN` and turns a wrong proof into a SystemError (#148) | - | `array.array[float]` / `array.array[int]` parameters (exact `array.array`, typecode `d` or `q`/`l`, 8-byte items) are read and written in place: the buffer export is held for the call and released on every exit, and every element access is one relaxed atomic 8-byte load or store, so each access is atomic like the interpreter's `a[i]`; no copy-in and no limit on the number of buffers. Open (#149): while the export is held, another thread's resize of that array raises BufferError. numpy arrays are not compiled yet (#220) | anything else: object path | + | `array.array[float]` / `array.array[int]` parameters (exact `array.array`, typecode `d` or `q`/`l`, 8-byte items) are read and written in place: the buffer export is held for the call and released on every exit, and every element access is one relaxed atomic 8-byte load or store, so each access is atomic like the interpreter's `a[i]`; no copy-in and no limit on the number of buffers. Open (#149): while the export is held, another thread's resize of that array raises BufferError. numpy arrays: see the next row | anything else: object path | + | numpy `npt.NDArray[np.float64]` / `NDArray[np.int64]` parameters (exact `numpy.ndarray`, 1-D, C-contiguous, native byte order, writable when stored to) take the same buffer path (#220, step 1) when the function accesses the array purely by element (`x[i]`, `len(x)`); a function that uses it as a whole object (`np.sum(x)`, `x.T`, passing `x` on, a slice) compiles as before with the parameter an opaque object. Other dtypes, including bool, are opaque objects. Results follow the interpreter: a value derived from an element is a numpy scalar there, so it is boxed as `np.float64` / `np.int64` / `np.bool` when it escapes (return, `Box`), decided per value at run time by a taint flag; a function declared `-> float` / `int` / `bool` that holds a numpy array returns the interpreter's object. `float(v)` of such a value is a plain float. An out-of-range index raises numpy's `IndexError("index i is out of bounds for axis 0 with size n")`. The numpy RuntimeWarnings (overflow, invalid value) and numpy's division are not emulated in this step: a pure function deopts on a non-finite float `+ - *` result or an unproved int overflow, so the interpreter emits them; `/`, `//`, `%` on a value derived from an element, and `+ - *` on one in an impure function, are not compiled | anything else: object path | | A function holding an array parameter runs no user code: closed, and no slot read or `C(args)` whose slow path runs user code, in its whole call tree (#203) | not compiled; diagnostic | | OBJ references follow one ownership rule; no IR form for free, borrowed alias or NULL read | not compiled; diagnostic | | Integer overflow checked and promoted (deopt); an op marked `proven` must re-prove its interval fits i64 | not compiled; diagnostic `verify/proven-overflow` | diff --git a/python-multiplatform-ksp/src/main/python/typedpython/cgen.py b/python-multiplatform-ksp/src/main/python/typedpython/cgen.py index 7b1ea1bf..7f3bf55c 100644 --- a/python-multiplatform-ksp/src/main/python/typedpython/cgen.py +++ b/python-multiplatform-ksp/src/main/python/typedpython/cgen.py @@ -37,12 +37,11 @@ (tp_runtime.h). Buffer arrays (issue #149, identity-safe part). A parameter annotated `array.array[float]`, -`array.array[int]` is an `ArrayParam` with `buffer` "array". numpy: not supported yet (#220): -a numpy element's arithmetic produces np.float64 / np.int64 / np.bool (compiled code returns float), -numpy scalar operations emit observable RuntimeWarnings and divide by zero without raising (compiled -double arithmetic does neither), and emulating that needs taint tracking of numpy-derived values, -numpy-scalar boxing and FP-exception-to-warning emulation. An `NDArray[...]` parameter is an ordinary -object parameter. `tp_wrap_f` does not copy it: after the scalar guards, `tp_buf_enter` +`array.array[int]` is an `ArrayParam` with `buffer` "array"; `NDArray[np.float64]` / `NDArray[np.int64]` +(#220) is one with `buffer` "numpy" (exact `numpy.ndarray`, an out-of-range index raises numpy's +IndexError text). A numpy element is an np.float64 / np.int64 in the interpreter: `Box.np` boxes +the matching numpy scalar when its run-time flag is true, and `BinOp.np` deopts a non-finite float +`+ - *` result so the interpreter emits numpy's RuntimeWarning (taint.py). `tp_wrap_f` does not copy it: after the scalar guards, `tp_buf_enter` checks the exact type (captured in `tp_state.buf_types` by the exec slot), takes one `Py_buffer` export (C-contiguous, 1-D, 8-byte items, format `d` / `q` / `l`, aligned, writable when the function stores) and any mismatch deopts before an effect. The export is released by `tp_buf_exit`, part of the @@ -186,7 +185,9 @@ class CGenError(Exception): _ARRAY_CT = {Type.F64_ARRAY: "tp_f64_array", Type.I64_ARRAY: "tp_i64_array"} # Buffer arrays (#149): `ArrayParam.buffer` -> the exact type the guard compares against (the index # into the module state's `buf_types`), and the format strings the export must carry. -_BUFFER_KINDS = {None: None, "array": 0} +_BUFFER_KINDS = {None: None, "array": 0, "numpy": 1} +# buf_types slots 2..4: the numpy scalar types a Box.np makes (np.float64, np.int64, np.bool_) +_NP_SLOT = {Type.F64: 2, Type.I64: 3, Type.BOOL: 4} _BUFFER_FMT = {Type.F64_ARRAY: ('"d"', "NULL"), Type.I64_ARRAY: ('"q"', '"l"')} _BUFFER_SFX = {Type.F64_ARRAY: "f64", Type.I64_ARRAY: "i64"} _CT_ARRAY_REV = {"tp_f64_array": Type.F64_ARRAY, "tp_i64_array": Type.I64_ARRAY} @@ -672,7 +673,7 @@ def generate(module: ir.Module, source_path: Path, display_path: str | None = No w(f" PyObject *ints[{len(mg.int_consts) + 1}]; /* strong: the int constants boxed once (SPEC N-13) */") w(f" tp_gslot gc[{len(mg.gnames) + 1}]; /* cached global/builtin reads (SPEC N-13) */") w(" PyObject *builtins; /* strong: the builtins dict the module source ran with; NULL: none */") - w(" PyObject *buf_types[1]; /* strong: array.array (buffer arrays, #149); NULL: not importable */") + w(" PyObject *buf_types[5]; /* strong: array.array, numpy.ndarray, np.float64, np.int64, np.bool_ (buffer arrays #149, #220); NULL: not importable */") w(" PyObject *mtype; /* strong: the descriptor type of compiled methods; NULL: none */") w(f" PyObject *mdescrs[{mg.n_methods + 1}]; /* strong: the installed descriptor of each method */") w("} tp_state;") @@ -1048,6 +1049,9 @@ def ev_BinOp(self, e: ir.BinOp) -> str: t = self.tmp("double") if e.op in _F64_INLINE: self.emit(f"{t} = {a} {_F64_INLINE[e.op]} {b};") + if e.np: + # #220: numpy would warn (overflow / invalid value); the interpreter redoes the call + self.emit(f"if (!isfinite({t})) {{ tp_rc = 1; goto tp_exit; }}") else: self.check(f"{_F64_OP[e.op]}({a}, {b}, &{t})") return t @@ -1155,6 +1159,8 @@ def ev_Truth(self, e: ir.Truth) -> str: return t def ev_ToFloat(self, e: ir.ToFloat) -> str: + if e.explicit and e.operand.type is Type.F64: + return self.ev(e.operand) if e.operand.type is not Type.I64: raise CGenError(f"{self.f.name}: ToFloat of {e.operand.type}") v = self.ev(e.operand) @@ -1194,6 +1200,12 @@ def ev_Box(self, e: ir.Box) -> str: if ot not in _BOX: raise CGenError(f"{self.f.name}: Box of {ot}") v = self.ev(e.operand) + if e.np is not None: + if e.np.type is not Type.BOOL: + raise CGenError(f"{self.f.name}: Box np flag of {e.np.type}") + flag = self.ev(e.np) + return self._own_new(f"({flag} ? tp_box_np(tp_st->buf_types[{_NP_SLOT[ot]}], " + f"{_BOX[ot]}({v})) : {_BOX[ot]}({v}))") return self._own_new(f"{_BOX[ot]}({v})") def ev_Unbox(self, e: ir.Unbox) -> str: @@ -2108,7 +2120,8 @@ def wrapper(self) -> str: for i, p in buffers: fmt1, fmt2 = _BUFFER_FMT[p.type] kind = _BUFFER_KINDS[p.buffer] - L.append(f" tp_s = tp_buf_enter(tp_args[{i}], " + enter = "tp_buf_enter_np" if p.buffer == "numpy" else "tp_buf_enter" + L.append(f" tp_s = {enter}(tp_args[{i}], " f"((tp_state *)PyModule_GetState(tp_module))->buf_types[{kind}], " f"{fmt1}, {fmt2}, {int(p.stored)}, &tp_a{i});") L.append(" if (tp_s < 0) goto tp_error;") @@ -2403,7 +2416,7 @@ def _state_functions(mg: _ModGen) -> str: " for (i = 0; i < sizeof(tp_st->ints) / sizeof(tp_st->ints[0]); i++) Py_VISIT(tp_st->ints[i]);", " Py_VISIT(tp_st->mtype);", " for (i = 0; i < sizeof(tp_st->mdescrs) / sizeof(tp_st->mdescrs[0]); i++) Py_VISIT(tp_st->mdescrs[i]);", - " for (i = 0; i < 1; i++) Py_VISIT(tp_st->buf_types[i]);", + " for (i = 0; i < sizeof(tp_st->buf_types) / sizeof(tp_st->buf_types[0]); i++) Py_VISIT(tp_st->buf_types[i]);", " for (i = 0; i < sizeof(tp_st->names) / sizeof(tp_st->names[0]); i++) Py_VISIT(tp_st->names[i]);", " for (i = 0; i < sizeof(tp_st->kwnames) / sizeof(tp_st->kwnames[0]); i++) Py_VISIT(tp_st->kwnames[i]);", " for (i = 0; i < sizeof(tp_st->classes) / sizeof(tp_st->classes[0]); i++) Py_VISIT(tp_st->classes[i]);", @@ -2425,7 +2438,7 @@ def _state_functions(mg: _ModGen) -> str: " for (i = 0; i < sizeof(tp_st->gc) / sizeof(tp_st->gc[0]); i++) tp_gcache_clear(&tp_st->gc[i]);", " for (i = 0; i < sizeof(tp_st->mdescrs) / sizeof(tp_st->mdescrs[0]); i++) tp_release(&tp_st->mdescrs[i]);", " tp_release(&tp_st->mtype);", - " for (i = 0; i < 1; i++) tp_release(&tp_st->buf_types[i]);", + " for (i = 0; i < sizeof(tp_st->buf_types) / sizeof(tp_st->buf_types[0]); i++) tp_release(&tp_st->buf_types[i]);", " for (i = 0; i < sizeof(tp_st->names) / sizeof(tp_st->names[0]); i++) tp_release(&tp_st->names[i]);", " for (i = 0; i < sizeof(tp_st->kwnames) / sizeof(tp_st->kwnames[0]); i++) tp_release(&tp_st->kwnames[i]);", " for (i = 0; i < sizeof(tp_st->cls_ok) / sizeof(tp_st->cls_ok[0]); i++) tp_st->cls_ok[i] = 0;", @@ -2541,6 +2554,9 @@ def _exec_function(mg: _ModGen) -> str: # module source ran (so its own imports are done). Not importable: NULL, and every guard deopts. if "array" in kinds: L.append(' if (tp_import_attr("array", "array", &tp_st->buf_types[0]) < 0) goto tp_done;') + if "numpy" in kinds: + for slot, attr in ((1, "ndarray"), (2, "float64"), (3, "int64"), (4, "bool_")): + L.append(f' if (tp_import_attr("numpy", "{attr}", &tp_st->buf_types[{slot}]) < 0) goto tp_done;') L += [ " tp_table = PyDict_New();", " if (tp_table == NULL || PyDict_SetItemString(tp_dict, \"__typedpython_interpreted__\", tp_table) < 0) goto tp_done;", diff --git a/python-multiplatform-ksp/src/main/python/typedpython/frontend.py b/python-multiplatform-ksp/src/main/python/typedpython/frontend.py index c9956b33..d34fc22b 100644 --- a/python-multiplatform-ksp/src/main/python/typedpython/frontend.py +++ b/python-multiplatform-ksp/src/main/python/typedpython/frontend.py @@ -77,7 +77,7 @@ from dataclasses import dataclass, field from pathlib import Path -from typedpython import gate, ir, pyrefly, rebinding +from typedpython import gate, ir, pyrefly, rebinding, taint from typedpython.ir import Type MARKER = "# typedpython: compiled" @@ -315,9 +315,35 @@ def _buffer_kind(annotation: ast.expr, info: _ModuleInfo) -> tuple[Type, str] | if base == "array.array" and isinstance(arg, ast.Name) and arg.id in ("float", "int") \ and not info.bound(arg.id): return ARRAY_KIND[f"list[{arg.id}]"], "array" + if base == "numpy.typing.NDArray": + dtype = _qualified(arg, info) + if dtype in _NUMPY_DTYPE: + return _NUMPY_DTYPE[dtype], "numpy" return None +def _element_only(fn: ast.FunctionDef, name: str) -> bool: + """Whether every use of `name` in `fn` is `name[i]` (read or store, not a slice) or `len(name)`, + and it is never rebound: the only uses a buffer array supports.""" + ok: set[int] = set() + for n in ast.walk(fn): + if isinstance(n, ast.Subscript) and isinstance(n.value, ast.Name) and n.value.id == name \ + and not isinstance(n.slice, ast.Slice) \ + and not (isinstance(n.slice, ast.Tuple) and any(isinstance(e, ast.Slice) for e in n.slice.elts)): + ok.add(id(n.value)) + elif isinstance(n, ast.Call) and isinstance(n.func, ast.Name) and n.func.id == "len" \ + and len(n.args) == 1 and isinstance(n.args[0], ast.Name) and n.args[0].id == name: + ok.add(id(n.args[0])) + return all(id(n) in ok for n in ast.walk(fn) + if isinstance(n, ast.Name) and n.id == name and not isinstance(n.ctx, ast.Param) + ) and not any(isinstance(n, ast.Name) and n.id == name and isinstance(n.ctx, ast.Store) + for n in ast.walk(fn)) + + +# numpy dtypes that are 8-byte buffer arrays (#220): the scalar type annotation they spell +_NUMPY_DTYPE = {"numpy.float64": Type.F64_ARRAY, "numpy.int64": Type.I64_ARRAY} + + def _scalar_globals(tree: ast.Module, bindings: dict[str, list[tuple[int, str]]]) -> dict[str, Type]: """Module-level names that are assigned exactly once, at top level, with a float/int/bool value, and are never rebound (`global`, `del`, `:=`, a loop target ...) anywhere in the module.""" @@ -650,6 +676,8 @@ def _signature(fn: ast.FunctionDef, classes: dict[str, ir.ClassDecl] | None = No raise Skip(a.lineno, f"parameter `{a.arg}` has no annotation") kind = _annotation_kind(a.annotation) buffer = _buffer_kind(a.annotation, info) if info is not None else None + if buffer is not None and buffer[1] == "numpy" and not _element_only(fn, a.arg): + buffer = None # #220: used as a whole object (np.sum(x), x.T, passed on): opaque if buffer is not None: params.append(ir.ArrayParam(a.arg, buffer[0], False, buffer[1])) # `stored` set later elif kind in ARRAY_KIND: @@ -671,6 +699,9 @@ def _signature(fn: ast.FunctionDef, classes: dict[str, ir.ClassDecl] | None = No returns = Type.NONE else: returns = SCALAR_KIND.get(_annotation_kind(fn.returns), Type.OBJ) + if returns in ir.SCALARS and any( + isinstance(p, ir.ArrayParam) and p.buffer == "numpy" for p in params): + returns = Type.OBJ # #220: the interpreter returns np.float64 / np.int64 / np.bool elements, ellipsis = _tuple_annotation(fn.returns) return _Signature(fn, tuple(params), returns, elements, ellipsis, _src(fn.returns), owner) @@ -733,6 +764,8 @@ def _lower_function( kinds: dict[str, dict[str, list[tuple[int, str]]]], ) -> ir.Function | str: inferred = kinds.get(sig.qname, {}) + if any(isinstance(p, ir.ArrayParam) and p.buffer == "numpy" for p in sig.params): + inferred = _numpy_kinds(sig, inferred) pure_error: Skip | None = None try: fn = _lower_in_mode(sig, env, info, inferred, impure=False) @@ -751,6 +784,65 @@ def _lower_function( return e.reason +_NUMPY_SPELLED = {"float64": "float", "numpy.float64": "float", "int64": "int", "numpy.int64": "int", + "bool_": "bool", "numpy.bool_": "bool", "numpy.bool": "bool"} + + +def _numpy_kinds(sig: _Signature, inferred: dict[str, list[tuple[int, str]]], + ) -> dict[str, list[tuple[int, str]]]: + """Pyrefly spells a value computed from a numpy element `Unknown` when it cannot import numpy + (or `float64` / `int64` when it can): in a function with numpy arrays (#220) those entries are + re-read from the source, as the scalar kind (float / int / bool) the interpreter's numpy scalar + stands for. An entry stays `Unknown` when the source does not settle it.""" + arrays = {p.name: p.type for p in sig.params if isinstance(p, ir.ArrayParam) and p.buffer == "numpy"} + scalars = {p.name: _annotation_kind(p_node.annotation) + for p, p_node in zip(sig.params, sig.node.args.args) + if isinstance(p, ir.Param) and p.type in ir.SCALARS} + nodes = {n.lineno: n for n in ast.walk(sig.node) + if isinstance(n, (ast.Assign, ast.AugAssign)) and len(getattr(n, "targets", [0])) == 1} + + out = {name: [(line, _NUMPY_SPELLED.get(k, k)) for line, k in entries] + for name, entries in inferred.items()} + + def known(name: str) -> str | None: + ks = {k for _, k in out.get(name, []) if k != "Unknown"} + if name in scalars: + ks.add(scalars[name]) + return next(iter(ks)) if len(ks) == 1 else None + + def kind(e: ast.expr) -> str | None: + if isinstance(e, ast.Constant): + return {bool: "bool", int: "int", float: "float"}.get(type(e.value)) + if isinstance(e, ast.Name): + return known(e.id) + if isinstance(e, ast.Subscript) and isinstance(e.value, ast.Name) and e.value.id in arrays: + return "float" if arrays[e.value.id] is Type.F64_ARRAY else "int" + if isinstance(e, ast.UnaryOp): + return "bool" if isinstance(e.op, ast.Not) else kind(e.operand) + if isinstance(e, ast.BinOp): + a, b = kind(e.left), kind(e.right) + if a is None or b is None or "bool" in (a, b): + return None + return "float" if "float" in (a, b) or isinstance(e.op, ast.Div) else "int" + if isinstance(e, ast.Compare): + return "bool" + return None + + for _ in range(3): + for name, entries in out.items(): + fixed = [] + for line, k in entries: + node = nodes.get(line) + if k == "Unknown" and node is not None: + value = node.value + if isinstance(node, ast.AugAssign): + value = ast.BinOp(left=ast.Name(id=name), op=node.op, right=node.value) + k = kind(value) or k + fixed.append((line, k)) + out[name] = fixed + return out + + def _lower_in_mode(sig, env, info, inferred, impure: bool) -> ir.Function: fn = _lower_with_bounds(sig, env, info, inferred, impure, info.scalar_globals) if fn.entry_globals and not _closed(fn, env): @@ -857,6 +949,8 @@ def __init__(self, sig: _Signature, env: dict[str, ir.Function], info: _ModuleIn self.params = {p.name: p for p in sig.params} self.arrays = {p.name: p.type for p in sig.params if isinstance(p, ir.ArrayParam)} + self.numpy = {p.name for p in sig.params + if isinstance(p, ir.ArrayParam) and p.buffer == "numpy"} self.bindings: dict[str, list[_Binding]] = {} self.annotations: dict[str, list[tuple[int, str]]] = {} self._collect_bindings() @@ -909,6 +1003,15 @@ def run(self) -> ir.Function: if name not in self.params: locals_[name] = self.local_type(name, self.bindings[name][0].line) locals_.update(self.temps) + if self.numpy: + try: + body, flags, deopts = taint.apply( + body, self.numpy, not self.impure, + frozenset(p.name for p in self.sig.params if isinstance(p, ir.Param))) + except taint.Refuse as why: + raise Skip(self.fn.lineno, str(why)) from None + locals_.update(flags) + self.deopts = self.deopts or deopts params = tuple( ir.ArrayParam(p.name, p.type, p.name in self.stored, p.buffer) if isinstance(p, ir.ArrayParam) @@ -1895,6 +1998,9 @@ def call(self, node: ast.Call) -> ir.Expr: return ir.Len(Type.I64, node.args[0].id) if plain and len(node.args) == 1 and name == "float" and self.builtin("float"): arg = self.value(node.args[0]) + if self.numpy and arg.type in (Type.I64, Type.F64): + # #220: float(np.float64) / float(np.int64) is a plain float + return ir.ToFloat(Type.F64, arg, explicit=True) if arg.type == Type.I64: return ir.ToFloat(Type.F64, arg) if arg.type == Type.F64: diff --git a/python-multiplatform-ksp/src/main/python/typedpython/ir.py b/python-multiplatform-ksp/src/main/python/typedpython/ir.py index d4163030..f337e09f 100644 --- a/python-multiplatform-ksp/src/main/python/typedpython/ir.py +++ b/python-multiplatform-ksp/src/main/python/typedpython/ir.py @@ -157,6 +157,10 @@ class BinOp(Expr): # Any other op, or an op the verifier cannot prove, is rejected (verify/proven-overflow). proven: bool = False inplace: bool = False + # F64 ADD/SUB/MUL only (#220): an operand may be a numpy scalar in the interpreter, which warns + # (RuntimeWarning: overflow / invalid value) where a double does not. A non-finite result → + # deopt, so the interpreter redoes the call and emits the warning; legal only in a pure function. + np: bool = False class UnaryOpKind(Enum): @@ -217,6 +221,9 @@ class ToFloat(Expr): int→float conversion of mixed arithmetic. Type F64.""" operand: Expr + # `float(x)` written in the source (#220): the result is a plain float even when `x` is a numpy + # scalar in the interpreter. The operand may then also be F64 (the value itself). + explicit: bool = False @dataclass(frozen=True) @@ -231,9 +238,14 @@ class Box(Expr): """A scalar or array element converted to a Python object (type OBJ): I64 → int, F64 → float, BOOL → True/False, INT → the int (a promoted INT is the PyLong it holds). Used when a typed value flows into OBJ code. A Box of an I64 or INT constant is read from module state, built once at - module exec (SPEC N-13).""" + module exec (SPEC N-13). + `np` (#220, a BOOL expression, None otherwise): when it is true at run time the value is a numpy + scalar in the interpreter (it derives from an element of a numpy array), and the box is the + matching `np.int64` / `np.float64` / `np.bool` instead. The front end computes it per value (a + shadow BOOL local per scalar local that can hold one).""" operand: Expr + np: Expr | None = None @dataclass(frozen=True) @@ -700,8 +712,10 @@ class ArrayParam: # copy-in, no write-back, no critical section, no aliasing guard (two buffers may share memory # and reads then see writes in program order, as in the interpreter). `len` is the export's # shape[0], fixed for the call because the export forbids resizing. Closedness (#203) applies - # unchanged. numpy.ndarray is not a buffer kind (its element arithmetic is numpy's: np.float64 - # results, RuntimeWarnings, no ZeroDivisionError); an ndarray parameter is an ordinary object. + # unchanged. "numpy" (#220): an exact `numpy.ndarray` with dtype float64 (F64_ARRAY) or int64 + # (I64_ARRAY), same guard and element access; an out-of-range index raises numpy's IndexError + # ("index i is out of bounds for axis 0 with size n"). The elements are numpy scalars in the + # interpreter, see Box.np and BinOp.np. # Every other rule above for lists (stored, array-open, Index/StoreIndex/Len only) holds too. buffer: str | None = None diff --git a/python-multiplatform-ksp/src/main/python/typedpython/runtime/tp_runtime.h b/python-multiplatform-ksp/src/main/python/typedpython/runtime/tp_runtime.h index e45eef64..22f7d580 100644 --- a/python-multiplatform-ksp/src/main/python/typedpython/runtime/tp_runtime.h +++ b/python-multiplatform-ksp/src/main/python/typedpython/runtime/tp_runtime.h @@ -663,6 +663,23 @@ static inline int tp_math_hypot(double x, double y, double *out) static inline PyObject *tp_box_i64(int64_t v) { return PyLong_FromLongLong((long long)v); } static inline PyObject *tp_box_f64(double v) { return PyFloat_FromDouble(v); } static inline PyObject *tp_box_bool(int v) { return PyBool_FromLong(v ? 1 : 0); } +/* The numpy scalar of type `t` (np.float64 / np.int64 / np.bool_) holding the Python value `o` (a new + * reference, consumed); NULL with an exception set. Box.np (#220). */ +static inline PyObject *tp_box_np(PyObject *t, PyObject *o) +{ + PyObject *r; + if (o == NULL) { + return NULL; + } + if (t == NULL) { + Py_DECREF(o); + PyErr_SetString(PyExc_SystemError, "typedpython: numpy scalar type is unavailable"); + return NULL; + } + r = PyObject_CallOneArg(t, o); + Py_DECREF(o); + return r; +} static inline int tp_unbox_i64(PyObject *o, int64_t *out) { @@ -899,7 +916,7 @@ static inline int tp_i64_array_exit(tp_i64_array *a) * cost, it is not a supported configuration for threads. len is the export's shape[0], fixed for * the call (the export forbids resizing). * ------------------------------------------------------------------------------------------ */ -typedef struct { Py_buffer view; Py_ssize_t len; void *data; int held; } tp_buf; +typedef struct { Py_buffer view; Py_ssize_t len; void *data; int held; int np; } tp_buf; /* `mod.attr` as a new reference in *out; not importable (or no such attribute): *out stays NULL * and the error is cleared, so every guard against it deopts. -1 only for MemoryError. */ @@ -962,10 +979,23 @@ static inline int tp_buf_enter(PyObject *obj, PyObject *exact, const char *fmt1, return 0; } +/* A numpy array (#220): the same guard on the exact `numpy.ndarray`; `np` selects numpy's IndexError + * text. The format check already refuses a byte-order prefix and every dtype but float64 / int64. */ +static inline int tp_buf_enter_np(PyObject *obj, PyObject *exact, const char *fmt1, const char *fmt2, + int writable, tp_buf *b) +{ + int rc = tp_buf_enter(obj, exact, fmt1, fmt2, writable, b); + if (rc == 0) { + b->np = 1; + } + return rc; +} + /* index: a negative index counts from the end; out of range -> IndexError with the text CPython's * own array.array uses ("array index out of range" / "array assignment index out of range") */ static inline int tp_buf_slot(const tp_buf *b, int64_t i, int store, Py_ssize_t *slot) { + int64_t orig = i; if (i < 0) { i += (int64_t)b->len; if (i < 0) goto out_of_range; @@ -976,6 +1006,11 @@ static inline int tp_buf_slot(const tp_buf *b, int64_t i, int store, Py_ssize_t *slot = (Py_ssize_t)i; return 0; out_of_range: + if (b->np) { + PyErr_Format(PyExc_IndexError, "index %lld is out of bounds for axis 0 with size %zd", + (long long)orig, b->len); + return -1; + } PyErr_SetString(PyExc_IndexError, store ? "array assignment index out of range" : "array index out of range"); return -1; diff --git a/python-multiplatform-ksp/src/main/python/typedpython/taint.py b/python-multiplatform-ksp/src/main/python/typedpython/taint.py new file mode 100644 index 00000000..581dfaa1 --- /dev/null +++ b/python-multiplatform-ksp/src/main/python/typedpython/taint.py @@ -0,0 +1,183 @@ +"""numpy taint (issue #220, step 1): which scalar values are numpy scalars in the interpreter. + +In the interpreter an element of a numpy array is an `np.float64` / `np.int64`, and so is every value +computed from one with arithmetic (`s += x[i]`, `x[i] > 0` is an `np.bool`). Compiled code computes +with C scalars, so the front end records, per scalar local, a shadow BOOL local `__np` that says +whether the value held is such a numpy scalar, and every `Box` of a scalar carries the flag +(`Box.np`) so the object that escapes (return value, argument of a call, tuple element...) has the +interpreter's type. The flag is a run-time value because it can differ per path (`s = 0.0` is a +float until the loop adds an element). + +Conservative rules (decision 2026-10-25): whatever the pass cannot reproduce exactly is refused +(`Refuse`, the function stays interpreted): + * `/`, `//`, `%` with a tainted operand (numpy divides without raising and warns), + * `+ - *` on F64 with a tainted operand in an impure function (numpy warns on overflow / invalid + value; a pure function deopts on a non-finite result instead, `BinOp.np`), + * a tainted value stored into a list or local array, passed to a compiled callee, or used as the + fill of a local array (the object stored or passed would have to be the numpy scalar), + * a comparison of an I64 with an F64 where one side is tainted (numpy converts the int to double, + CPython compares exactly), `And` / `Or` of values that are not all plain. +What this cannot find: a numpy semantic not listed above that the front end lowers to an IR node +(there is none today for + - * unary, compares, math.*, len, index); the tests pin each. +""" +from __future__ import annotations + +import dataclasses + +from . import ir +from .ir import Type + + +class Refuse(Exception): + """The function uses numpy-derived values in a way step 1 does not reproduce.""" + + +TRUE = ir.Const(Type.BOOL, True) + + +def flag_name(name: str) -> str: + return f"{name}__np" + + +def _or(a: ir.Expr | None, b: ir.Expr | None) -> ir.Expr | None: + if a is None: + return b + if b is None: + return a + if a == TRUE or b == TRUE: + return TRUE + return ir.Or(Type.BOOL, a, b) + + +class _Pass: + def __init__(self, numpy: set[str], pure: bool) -> None: + self.numpy = numpy + self.pure = pure + self.flagged: set[str] = set() + self.deopts = False # some BinOp got `np` (it can deopt) + + # --- taint of an expression: None (a plain value) or a BOOL expression --------------------- + + def taint(self, e: ir.Expr) -> ir.Expr | None: + if e.type not in ir.SCALARS: + return None + if isinstance(e, ir.Index): + return TRUE if e.array in self.numpy else None + if isinstance(e, ir.Local): + return ir.Local(Type.BOOL, flag_name(e.name)) if e.name in self.flagged else None + if isinstance(e, ir.ToFloat): + return None if e.explicit else self.taint(e.operand) + if isinstance(e, ir.BinOp): + t = _or(self.taint(e.left), self.taint(e.right)) + if t is not None and e.op in (ir.BinOpKind.TRUEDIV, ir.BinOpKind.FLOORDIV, + ir.BinOpKind.MOD): + raise Refuse(f"`{e.op.value}` of a value derived from a numpy element (numpy divides " + "without raising and warns; not reproduced yet)") + if t is not None and e.type is Type.F64 and not self.pure: + raise Refuse(f"`{e.op.value}` of a value derived from a numpy element in a function " + "with effects (numpy would warn on overflow; not reproduced yet)") + return t + if isinstance(e, ir.UnaryOp): + return None if e.op is ir.UnaryOpKind.NOT else self.taint(e.operand) + if isinstance(e, ir.Compare): + t = _or(self.taint(e.left), self.taint(e.right)) + if t is not None and e.left.type is not e.right.type and Type.BOOL not in ( + e.left.type, e.right.type): + raise Refuse("comparison of an int and a float where one is derived from a numpy " + "element (numpy converts the int to double, CPython compares exactly)") + return t + if isinstance(e, (ir.And, ir.Or)): + if self.taint(e.left) is not None or self.taint(e.right) is not None: + raise Refuse("`and` / `or` of a value derived from a numpy element") + return None + if isinstance(e, ir.Call): + if any(self.taint(a) is not None for a in e.args): + raise Refuse(f"a value derived from a numpy element is passed to `{e.function}`") + return None + # Const, Len, MathCall (a float), Unbox, ObjToFloat, ...: plain values + return None + + def plain(self, e: ir.Expr, what: str) -> None: + if self.taint(e) is not None: + raise Refuse(f"a value derived from a numpy element is {what}") + + # --- discovery of the locals that need a flag --------------------------------------------------- + + def discover(self, body) -> None: + while True: + before = len(self.flagged) + self._scan(body) + if len(self.flagged) == before: + return + + def _scan(self, x) -> None: + if isinstance(x, (tuple, list)): + for y in x: + self._scan(y) + elif isinstance(x, ir.Assign): + if x.value.type in ir.SCALARS and self.taint(x.value) is not None: + self.flagged.add(x.target) + elif dataclasses.is_dataclass(x) and not isinstance(x, type): + for f in dataclasses.fields(x): + self._scan(getattr(x, f.name)) + + # --- rewrite ----------------------------------------------------------------------------------- + + def rewrite(self, x): + if isinstance(x, tuple): + out = [] + for y in x: + r = self.rewrite(y) + out.extend(r if isinstance(r, list) else [r]) + return tuple(out) + if not dataclasses.is_dataclass(x) or isinstance(x, type): + return x + if isinstance(x, ir.Assign): + value = self.rewrite(x.value) + stmt = dataclasses.replace(x, value=value) + if x.target in self.flagged: + t = self.taint(x.value) + return [stmt, ir.Assign(flag_name(x.target), t if t is not None + else ir.Const(Type.BOOL, False))] + return stmt + if isinstance(x, ir.Box) and x.operand.type in ir.SCALARS: + return dataclasses.replace(x, operand=self.rewrite(x.operand), np=self.taint(x.operand)) + if isinstance(x, ir.BinOp) and x.type in ir.SCALARS: + self.taint(x) # refuses division / impure arithmetic wherever it appears + if isinstance(x, ir.BinOp) and x.type is Type.F64 and x.op in ( + ir.BinOpKind.ADD, ir.BinOpKind.SUB, ir.BinOpKind.MUL): + tainted = _or(self.taint(x.left), self.taint(x.right)) is not None + self.deopts = self.deopts or tainted + return dataclasses.replace(x, left=self.rewrite(x.left), right=self.rewrite(x.right), + np=tainted) + if isinstance(x, ir.StoreIndex) and x.array not in self.numpy: + self.plain(x.value, "stored into a list or local array") + if isinstance(x, ir.NewArray) and x.fill is not None: + self.plain(x.fill, "the fill of a local array") + if isinstance(x, ir.ToInt): + self.plain(x.operand, "widened to a native INT (SPEC N-13), which has no numpy form") + if isinstance(x, ir.Call): + self.taint(x) # refuses a tainted argument + if isinstance(x, ir.Compare): + self.taint(x) # refuses what cannot be reproduced even inside a condition + changes = {} + for f in dataclasses.fields(x): + v = getattr(x, f.name) + r = self.rewrite(v) + if r is not v: + changes[f.name] = r + return dataclasses.replace(x, **changes) if changes else x + + +def apply(body: tuple, numpy: set[str], pure: bool, + scalar_params: frozenset[str] = frozenset()) -> tuple[tuple, dict[str, Type], bool]: + """Rewrite a lowered body: shadow flags and `Box.np` / `BinOp.np`. Returns the new body and the + new locals (the flags, BOOL) and whether the body now can deopt. Raises Refuse.""" + p = _Pass(numpy, pure) + p.discover(body) + new = p.rewrite(body) + # a parameter that is reassigned from a numpy element: its flag starts False + first = tuple(ir.Assign(flag_name(n), ir.Const(Type.BOOL, False)) + for n in sorted(p.flagged & scalar_params)) + new = first + new + return new, {flag_name(n): Type.BOOL for n in p.flagged}, p.deopts diff --git a/python-multiplatform-ksp/src/main/python/typedpython/verify.py b/python-multiplatform-ksp/src/main/python/typedpython/verify.py index 99b25611..db8487d3 100644 --- a/python-multiplatform-ksp/src/main/python/typedpython/verify.py +++ b/python-multiplatform-ksp/src/main/python/typedpython/verify.py @@ -394,7 +394,7 @@ def declarations(self) -> frozenset[str]: self.error(TYPE, f"array parameter '{p.name}' has non-array type {p.type}") elif not isinstance(p.stored, bool): self.error(TYPE, f"array parameter '{p.name}' has a non-bool `stored`") - elif p.buffer not in (None, "array"): + elif p.buffer not in (None, "array", "numpy"): self.error(BUFFER, f"array parameter '{p.name}' has buffer kind {p.buffer!r}") else: self.arrays[p.name] = p.type @@ -771,6 +771,14 @@ def e_BinOp(self, e: BinOp, state: _State) -> ir.Expr: exact = (_exact(e.op, self.interval(left), self.interval(right)) if shape else _FULL) proven = self.check_proven(shape, exact, f"{e.type.value} {e.op.value}") + if type(e.np) is not bool: + self.error(TYPE, f"BinOp `np` {e.np!r} is not a bool") + elif e.np: + if not (e.type is Type.F64 and lt is Type.F64 and rt is Type.F64 + and e.op in (BinOpKind.ADD, BinOpKind.SUB, BinOpKind.MUL)): + self.error(TYPE, "BinOp `np` is set on something other than an f64 + - * (#220)") + else: + self.can_deopt = True # a non-finite result: the interpreter redoes the call shift = e.op in (BinOpKind.LSHIFT, BinOpKind.RSHIFT) if lt is Type.INT: # SPEC N-13: both operands INT, result INT, never deopts (it promotes in place) @@ -879,7 +887,10 @@ def _bool_pair(self, e: And | Or, state: _State, cond: bool = False) -> ir.Expr: def e_ToFloat(self, e: ToFloat, state: _State) -> ir.Expr: operand = self.expr(e.operand, state) - self.slot(operand, Type.I64, "operand of ToFloat") + if type(e.explicit) is not bool: + self.error(TYPE, f"ToFloat `explicit` {e.explicit!r} is not a bool") + self.slot(operand, Type.F64 if e.explicit and getattr(operand, "type", None) is Type.F64 + else Type.I64, "operand of ToFloat") if e.type is not Type.F64: self.error(TYPE, f"ToFloat gives f64, node says {e.type.value}") return dataclasses.replace(e, operand=operand) @@ -902,7 +913,13 @@ def e_Box(self, e: Box, state: _State) -> ir.Expr: self.error(TYPE, f"Box of {getattr(ot, 'value', ot)}") if e.type is not Type.OBJ: self.error(TYPE, f"Box gives obj, node says {e.type.value}") - return dataclasses.replace(e, operand=operand) + np_flag = e.np + if np_flag is not None: + np_flag = self.expr(e.np, state) + if ot not in ir.SCALARS: + self.error(TYPE, "Box `np` is set on a Box of something other than a scalar") + self.slot(np_flag, Type.BOOL, "`np` flag of Box") + return dataclasses.replace(e, operand=operand, np=np_flag) def e_Unbox(self, e: Unbox, state: _State) -> ir.Expr: operand = self.expr(e.operand, state) diff --git a/python-multiplatform-ksp/src/test/python/test_buffer_arrays.py b/python-multiplatform-ksp/src/test/python/test_buffer_arrays.py index 6fa1e44c..1c75f9ac 100644 --- a/python-multiplatform-ksp/src/test/python/test_buffer_arrays.py +++ b/python-multiplatform-ksp/src/test/python/test_buffer_arrays.py @@ -472,7 +472,7 @@ def test_verifier_buffer_kind_and_copy_rules(): from test_verify import F64 as VF64, F64A as VF64A, assert_rejected, fn from typedpython.ir import ArrayParam, Assign, Const, CopyArray, Index, Return, Type - for kind in ("numpy", "memoryview"): + for kind in ("ndarray", "memoryview"): # "numpy" is a buffer kind since #220 assert_rejected("verify/buffer", fn("f", params=[ArrayParam("a", VF64A, False, kind)], returns=VF64, pure=True, body=[Return(Index(VF64, "a", Const(Type.I64, 0)))])) @@ -480,6 +480,9 @@ def test_verifier_buffer_kind_and_copy_rules(): locals_={"b": VF64A}, body=[Assign("b", CopyArray(VF64A, "a")), Return(Index(VF64, "b", Const(Type.I64, 0)))]) assert_rejected("verify/buffer", copy) + import dataclasses + assert_rejected("verify/buffer", dataclasses.replace( + copy, params=[ArrayParam("a", VF64A, False, "numpy")])) def test_verifier_buffer_params_are_not_limited_but_lists_still_are(): diff --git a/python-multiplatform-ksp/src/test/python/test_buffer_numpy.py b/python-multiplatform-ksp/src/test/python/test_buffer_numpy.py index b03d49a8..30f3e681 100644 --- a/python-multiplatform-ksp/src/test/python/test_buffer_numpy.py +++ b/python-multiplatform-ksp/src/test/python/test_buffer_numpy.py @@ -1,17 +1,29 @@ -"""numpy arrays are not compiled yet (#220): an `NDArray[...]` parameter is an ordinary object -parameter, so results and their numpy scalar types equal the interpreter's. Skipped without numpy.""" +"""numpy array parameters on the zero-copy buffer path (issue #220, step 1; SPEC N-8 / N-9). + +An `NDArray[np.float64]` / `NDArray[np.int64]` parameter is read and written in place like an +`array.array`. What differs is what the interpreter computes: an element is a numpy scalar, so every +value derived from one is an `np.float64` / `np.int64` / `np.bool` when it escapes, numpy emits +RuntimeWarnings on overflow / invalid value, and numpy divides without raising. Step 1 reproduces +the result and its TYPE exactly (a run-time taint flag per local), and never differs silently on a +warning or a division: a pure function deopts (the interpreter then emits the warning), anything +else that could warn or divide is not compiled. Every test compares result type, value, emitted +warnings (`warnings.catch_warnings(record=True)`) and final array contents with the interpreted +function. Skipped without numpy. +""" import textwrap +import warnings import pytest np = pytest.importorskip("numpy") from typedpython import cbuild, ir # noqa: E402 -from test_buffer_arrays import AD, F, build_source, differential, load_so # noqa: E402 +from test_buffer_arrays import build_source, load_so # noqa: E402 SOURCE = textwrap.dedent(''' # typedpython: compiled import array + import math import numpy as np import numpy.typing as npt @@ -27,38 +39,442 @@ def nd_first(x: npt.NDArray[np.float64]) -> object: return x[0] - def nd_sum(x: npt.NDArray[np.float64]) -> object: + def nd_first_f(x: npt.NDArray[np.float64]) -> float: + return x[0] + + + def nd_first_i(x: npt.NDArray[np.int64]) -> int: + return x[0] + + + def nd_sum(x: npt.NDArray[np.float64]) -> float: s = 0.0 for i in range(len(x)): s += x[i] return s + + + def nd_sum_from_first(x: npt.NDArray[np.float64]) -> float: + s = x[0] + for i in range(1, len(x)): + s += x[i] + return s + + + def nd_isum(x: npt.NDArray[np.int64]) -> int: + s = 0 + for i in range(len(x)): + s += x[i] + return s + + + def nd_gt(x: npt.NDArray[np.float64]) -> object: + return x[0] > x[1] + + + def nd_flag(x: npt.NDArray[np.float64]) -> object: + b = x[0] > 0.0 + return b + + + def nd_plain(x: npt.NDArray[np.float64]) -> object: + return float(x[0]) + + + def nd_mixed(x: npt.NDArray[np.float64], k: float) -> object: + return k + 1.0 + + + def nd_sqrt(x: npt.NDArray[np.float64]) -> object: + return math.sqrt(x[0] * x[0]) + + + def nd_neg(x: npt.NDArray[np.float64]) -> object: + return -x[1] + + + def nd_len(x: npt.NDArray[np.float64]) -> int: + return len(x) + + + def nd_set(x: npt.NDArray[np.float64], i: int, v: float) -> None: + x[i] = v + + + def nd_copy(dst: npt.NDArray[np.float64], src: npt.NDArray[np.float64]) -> None: + for i in range(len(dst)): + dst[i] = src[i] + + + def nd_iset(x: npt.NDArray[np.int64], i: int, v: int) -> None: + x[i] = v + + + def nd_get(x: npt.NDArray[np.float64], i: int) -> object: + return x[i] + + + def nd_overflow(x: npt.NDArray[np.float64]) -> object: + return x[0] * x[1] + + + def nd_invalid(x: npt.NDArray[np.float64]) -> object: + return x[0] - x[1] + + + def nd_iover(x: npt.NDArray[np.int64]) -> object: + return x[0] + x[1] + + + def nd_energy(x: npt.NDArray[np.float64], y: npt.NDArray[np.float64], z: npt.NDArray[np.float64], + vx: npt.NDArray[np.float64], vy: npt.NDArray[np.float64], + vz: npt.NDArray[np.float64], m: npt.NDArray[np.float64]) -> float: + e = 0.0 + for i in range(len(m)): + e += 0.5 * m[i] * (vx[i] * vx[i] + vy[i] * vy[i] + vz[i] * vz[i]) + return e + + + def nd_copy7(x: npt.NDArray[np.float64], y: npt.NDArray[np.float64], z: npt.NDArray[np.float64], + vx: npt.NDArray[np.float64], vy: npt.NDArray[np.float64], + vz: npt.NDArray[np.float64], m: npt.NDArray[np.float64]) -> None: + for i in range(len(m)): + x[i] = vx[i] + y[i] = vy[i] + z[i] = vz[i] + m[i] = x[i] + + + def nd_div(x: npt.NDArray[np.float64]) -> object: + return x[0] / x[1] + + + def nd_idiv(x: npt.NDArray[np.int64]) -> object: + return x[0] // x[1] + + + def nd_param(x: npt.NDArray[np.float64], k: float, use: bool) -> object: + if use: + k = x[0] + return k + + + def nd_branch(x: npt.NDArray[np.float64], use: bool) -> object: + if use: + v = x[0] + else: + v = 2.5 + return v + + + def nd_tuple(x: npt.NDArray[np.int64]) -> tuple[int, int, bool]: + return (x[0], 3, x[1] > 0) + + + def nd_abs(x: npt.NDArray[np.float64]) -> object: + return abs(x[1]) + + + def nd_to_list(x: npt.NDArray[np.float64], dst: list[float]) -> None: + dst[0] = x[0] + + + def nd_mixed_cmp(x: npt.NDArray[np.int64], f: float) -> object: + if x[0] < f: + return 1 + return 0 + + + def whole_sum(x: npt.NDArray[np.float64]) -> object: + return np.sum(x) + + + def whole_attr(x: npt.NDArray[np.float64]) -> object: + return x.size + + + def whole_pass(x: npt.NDArray[np.float64]) -> object: + return np.add(x, 1.0).sum() + + + def nd_int_impure(x: npt.NDArray[np.int64]) -> object: + s = 0 + for i in range(len(x)): + s += x[i] + x[0] = 1 + return s + + + def nd_scale(x: npt.NDArray[np.float64], k: float) -> None: + for i in range(len(x)): + x[i] = x[i] * k ''') +COMPILED = { + "nd_first", "nd_first_f", "nd_first_i", "nd_sum", "nd_sum_from_first", "nd_isum", "nd_gt", + "nd_flag", "nd_plain", "nd_mixed", "nd_sqrt", "nd_neg", "nd_len", "nd_set", "nd_copy", + "nd_iset", "nd_get", "nd_param", "nd_branch", "nd_tuple", "nd_overflow", "nd_invalid", "nd_iover", "nd_energy", "nd_copy7", +} +NOT_COMPILED = {"nd_div", "nd_idiv", "nd_scale", "nd_to_list", "nd_mixed_cmp", "nd_abs", "nd_int_impure"} # nd_abs calls user code # could warn / divide, or store a possibly warning op + +F = [1.5, -2.25, 3.0, 4.5] +I = [3, -5, 7, 11] +BIG = 1.7e308 +IMAX = 2 ** 63 - 1 + @pytest.fixture(scope="module") def built(tmp_path_factory): d = tmp_path_factory.mktemp("bufnp") path, proved, diags, c = build_source(d, SOURCE) so = cbuild.build(c, path.stem, d / "out") - return proved, load_so(so) - - -def test_numpy_parameter_is_an_ordinary_object_and_keeps_numpy_results(built): - proved, m = built - for f in proved.functions: - assert all(not isinstance(p, ir.ArrayParam) or p.buffer == "array" or p.buffer is None - for p in f.params) - assert not (f.name.startswith("nd_") and any(isinstance(p, ir.ArrayParam) for p in f.params)) - x = np.array(F, dtype=np.float64) - r = m.nd_first(x) - assert type(r) is np.float64 and r == F[0] - r = m.nd_sum(x) - ref = 0.0 - for v in x: - ref += v - assert type(r) is np.float64 and r == ref - - -def test_a_numpy_array_given_to_an_array_parameter_takes_the_object_path(built): - _, m = built - differential(m, "sum_ad", lambda: (np.array(F),), expect_deopt=True) + return proved, load_so(so), c + + +def run(fn, args): + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter("always") + try: + r = ("ok", fn(*args)) + except Exception as e: # noqa: BLE001 - type and text are the observable + r = ("raise", type(e), str(e)) + return r, [(x.category, str(x.message), x.lineno) for x in w] + + +def same(a, b): + return a == b or (a != a and b != b) + + +def differential(m, name, make, expect_deopt=False): + ca, ia = make(), make() + before = m.__typedpython_deopts__ + c, cw = run(getattr(m, name), ca) + deopted = m.__typedpython_deopts__ - before + ref = m.__typedpython_interpreted__.get(name) or getattr(m, name) # not compiled: itself + i, iw = run(ref, ia) + assert c[0] == i[0], (c, i) + if c[0] == "ok": + assert type(c[1]) is type(i[1]), (type(c[1]), type(i[1])) + assert same(c[1], i[1]), (c, i) + else: + assert c[1:] == i[1:], (c, i) + assert cw == iw, (cw, iw) + for x, y in zip(ca, ia): + if isinstance(x, np.ndarray): + assert x.dtype == y.dtype and x.shape == y.shape + assert all(same(p, q) for p, q in zip(x.tolist(), y.tolist())), (x, y) + assert deopted == (1 if expect_deopt else 0), f"deopts: {deopted}, expected {expect_deopt}" + return c, cw + + +def f64(v=F): + return np.array(v, dtype=np.float64) + + +def i64(v=I): + return np.array(v, dtype=np.int64) + + +# --- what is compiled ------------------------------------------------------------------------------ + +def test_compiled_set(built): + proved, _, _ = built + names = {f.name for f in proved.functions} + assert COMPILED <= names, (sorted(COMPILED - names), proved.skipped) + for n in NOT_COMPILED: + assert n in proved.skipped, n + assert n not in names + + +def test_ir_marks_numpy_buffers(built): + proved, _, _ = built + by = {f.name: f for f in proved.functions} + assert [(p.buffer, p.stored) for p in by["nd_first"].params] == [("numpy", False)] + assert by["nd_set"].params[0].buffer == "numpy" and by["nd_set"].params[0].stored + assert by["nd_isum"].params[0].type is ir.Type.I64_ARRAY + assert by["nd_energy"].pure is True + assert sum(p.buffer == "numpy" for p in by["nd_energy"].params) == 7 + + +def test_seven_numpy_arrays_compile_and_run(built): + _, m, _ = built + arrs = lambda: tuple(f64([k + 0.5, k + 1.5, k + 2.5]) for k in range(7)) + c, _ = differential(m, "nd_energy", arrs) + assert type(c[1]) is np.float64 + differential(m, "nd_copy7", arrs) + + +def test_array_array_path_is_unchanged(built): + import array + _, m, _ = built + r = m.sum_ad(array.array("d", F)) + assert type(r) is float and r == sum(F) + + +# --- results and result types ------------------------------------------------------------------------ + +@pytest.mark.parametrize("name", ["nd_first", "nd_first_f"]) +def test_element_is_a_numpy_scalar(built, name): + _, m, _ = built + c, _ = differential(m, name, lambda: (f64(),)) + assert type(c[1]) is np.float64 + + +def test_int_element_is_np_int64_whatever_the_annotation_says(built): + _, m, _ = built + c, _ = differential(m, "nd_first_i", lambda: (i64(),)) + assert type(c[1]) is np.int64 + + +@pytest.mark.parametrize("name,make,t", [ + ("nd_sum", lambda: (f64(),), np.float64), + ("nd_sum", lambda: (f64([]),), float), # zero iterations: the plain 0.0 + ("nd_sum_from_first", lambda: (f64(),), np.float64), + ("nd_isum", lambda: (i64(),), np.int64), + ("nd_isum", lambda: (i64([]),), int), + ("nd_gt", lambda: (f64(),), np.bool_), + ("nd_flag", lambda: (f64(),), np.bool_), + ("nd_plain", lambda: (f64(),), float), # float(np.float64) is a plain float + ("nd_sqrt", lambda: (f64(),), float), # math.sqrt returns a float + ("nd_neg", lambda: (f64(),), np.float64), + ("nd_len", lambda: (f64(),), int), +]) +def test_result_type_follows_the_interpreter(built, name, make, t): + _, m, _ = built + c, _ = differential(m, name, make) + assert type(c[1]) is t + + +def test_plain_float_result_stays_float(built): + _, m, _ = built + c, _ = differential(m, "nd_mixed", lambda: (f64(), 2.0)) + assert type(c[1]) is float + + +def test_reads_stores_and_negative_indexes(built): + _, m, _ = built + for i in (0, 3, -1, -4): + differential(m, "nd_get", lambda: (f64(), i)) + differential(m, "nd_set", lambda: (f64(), i, 9.25)) + differential(m, "nd_iset", lambda: (i64(), i, 77)) + differential(m, "nd_copy", lambda: (f64([0.0] * 4), f64())) + + +@pytest.mark.parametrize("i", [4, -5, 100]) +def test_out_of_range_has_numpys_message(built, i): + _, m, _ = built + c, _ = differential(m, "nd_get", lambda: (f64(), i)) + assert c[0] == "raise" and c[1] is IndexError and "out of bounds for axis 0 with size 4" in c[2] + c, _ = differential(m, "nd_set", lambda: (f64(), i, 1.0)) + assert c[0] == "raise" and "out of bounds for axis 0 with size 4" in c[2] + + +def test_object_path_for_arrays_the_guard_refuses(built): + _, m, _ = built + x = f64(list(range(8))) + differential(m, "nd_first", lambda: (x[::2].copy(),)) # fine, contiguous + differential(m, "nd_first", lambda: (x[::2],), expect_deopt=True) # strided view + differential(m, "nd_first", lambda: (i64(),), expect_deopt=True) # wrong dtype + differential(m, "nd_first", lambda: (f64().astype(">f8"),), expect_deopt=True) + ro = f64() + ro.setflags(write=False) + differential(m, "nd_set", lambda: (ro, 0, 1.0), expect_deopt=True) # numpy's own error + differential(m, "nd_first", lambda: ([1.0, 2.0],), expect_deopt=True) + + +# --- warnings: a pure function deopts, the interpreter emits them ----------------------------------------- + +def test_float_overflow_warning_comes_from_the_interpreter(built): + _, m, _ = built + c, w = differential(m, "nd_overflow", lambda: (f64([BIG, 10.0]),), expect_deopt=True) + assert [x[0] for x in w] == [RuntimeWarning] and "overflow" in w[0][1] + differential(m, "nd_overflow", lambda: (f64([2.0, 10.0]),)) # no warning, no deopt + + +def test_invalid_warning_comes_from_the_interpreter(built): + _, m, _ = built + inf = float("inf") + c, w = differential(m, "nd_invalid", lambda: (f64([inf, inf]),), expect_deopt=True) + assert [x[0] for x in w] == [RuntimeWarning] and "invalid" in w[0][1] + + +def test_int_overflow_warning_comes_from_the_interpreter(built): + _, m, _ = built + c, w = differential(m, "nd_iover", lambda: (i64([IMAX, 1]),), expect_deopt=True) + assert [x[0] for x in w] == [RuntimeWarning] and "overflow" in w[0][1] + differential(m, "nd_iover", lambda: (i64(),)) + + +def test_division_and_impure_arithmetic_stay_on_the_object_path(built): + _, m, _ = built + for make in (lambda: (f64([1.0, 0.0]),), lambda: (f64([-1.0, 0.0]),), lambda: (f64([0.0, 0.0]),), + lambda: (f64(),)): + differential(m, "nd_div", make) + c, w = differential(m, "nd_idiv", lambda: (i64([5, 0]),)) + assert [x[0] for x in w] == [RuntimeWarning] + differential(m, "nd_scale", lambda: (f64([BIG, 1.0]), 10.0)) + differential(m, "nd_scale", lambda: (f64(), 2.0)) + + +def test_a_compiled_run_emits_no_warning_when_the_interpreter_emits_none(built): + _, m, _ = built + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter("always") + m.nd_sum(f64()) + m.nd_isum(i64()) + m.nd_energy(*(f64() for _ in range(7))) + assert w == [] + + +def test_taint_is_per_value_at_run_time(built): + _, m, _ = built + for use in (True, False): + c, _ = differential(m, "nd_param", lambda: (f64(), 7.0, use)) + assert type(c[1]) is (np.float64 if use else float) + c, _ = differential(m, "nd_branch", lambda: (f64(), use)) + assert type(c[1]) is (np.float64 if use else float) + + +def test_tuple_and_call_arguments_carry_numpy_scalars(built): + _, m, _ = built + c, _ = differential(m, "nd_tuple", lambda: (i64(),)) + assert [type(v) for v in c[1]] == [np.int64, int, np.bool_] + c, _ = differential(m, "nd_abs", lambda: (f64(),)) # interpreted: calls user code + assert type(c[1]) is np.float64 + + +def test_a_numpy_value_stored_into_a_list_or_compared_mixed_stays_interpreted(built): + _, m, _ = built + differential(m, "nd_to_list", lambda: (f64(), [0.0])) + differential(m, "nd_mixed_cmp", lambda: (i64([2 ** 53 + 1]), float(2 ** 53))) + + +@pytest.mark.parametrize("name", ["whole_sum", "whole_attr", "whole_pass"]) +def test_whole_array_functions_still_compile_with_an_opaque_parameter(built, name): + proved, m, _ = built + f = next((f for f in proved.functions if f.name == name), None) + assert f is not None, proved.skipped.get(name) + assert not any(isinstance(p, ir.ArrayParam) for p in f.params) + c, _ = differential(m, name, lambda: (f64(),)) + if name == "whole_sum": + assert type(c[1]) is np.float64 + + +def test_a_numpy_value_never_becomes_a_native_int(built): + proved, m, _ = built + assert "nd_int_impure" in proved.skipped + assert not any(type(n).__name__ == "ToInt" for f in proved.functions + if f.name.startswith("nd_") for n in _nodes(f.body)) + c, _ = differential(m, "nd_int_impure", lambda: (i64(),)) + assert type(c[1]) is np.int64 + + +def _nodes(x): + import dataclasses + if isinstance(x, (tuple, list)): + for y in x: + yield from _nodes(y) + elif dataclasses.is_dataclass(x) and not isinstance(x, type): + yield x + for f in dataclasses.fields(x): + yield from _nodes(getattr(x, f.name))