Skip to content

Feat: numpy array parameters, step 1 (#220) #263

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
b-re-w merged 2 commits into from
Oct 24, 2026
Merged

Feat: numpy array parameters, step 1 (#220) #263

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f8e90e3
WIP #220 step 1
b-re-w Oct 24, 2026
07e7b37
Merge develop into feat/tp-numpy: numpy params compose with INT (#220)
b-re-w Oct 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/SPEC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -688,7 +688,8 @@ All in `PM/commonTest`, so they run wherever the interpreter loads.
| Array access in bounds, or a runtime bounds check kept | the check stays (never an unchecked access) |
| Array parameters only indexed, stored, `len`-ed; no aliasing (entry guard) | not compiled; diagnostic |
| Integer ops proven safe by the verifier's intervals (constants, `len`, `range` variables, guards such as `if x < e` / `while lo < hi`, joined local assignments, `//` and `%` by constants) | emitted unchecked (a shift/mask for `//`/`%` by a power of two of a non-negative value); the sanitizer CI run builds with `TP_CHECK_PROVEN` and turns a wrong proof into a SystemError (#148) |
| `array.array[float]` / `array.array[int]` parameters (exact `array.array`, typecode `d` or `q`/`l`, 8-byte items) are read and written in place: the buffer export is held for the call and released on every exit, and every element access is one relaxed atomic 8-byte load or store, so each access is atomic like the interpreter's `a[i]`; no copy-in and no limit on the number of buffers. Open (#149): while the export is held, another thread's resize of that array raises BufferError. numpy arrays are not compiled yet (#220) | anything else: object path |
| `array.array[float]` / `array.array[int]` parameters (exact `array.array`, typecode `d` or `q`/`l`, 8-byte items) are read and written in place: the buffer export is held for the call and released on every exit, and every element access is one relaxed atomic 8-byte load or store, so each access is atomic like the interpreter's `a[i]`; no copy-in and no limit on the number of buffers. Open (#149): while the export is held, another thread's resize of that array raises BufferError. numpy arrays: see the next row | anything else: object path |
| numpy `npt.NDArray[np.float64]` / `NDArray[np.int64]` parameters (exact `numpy.ndarray`, 1-D, C-contiguous, native byte order, writable when stored to) take the same buffer path (#220, step 1) when the function accesses the array purely by element (`x[i]`, `len(x)`); a function that uses it as a whole object (`np.sum(x)`, `x.T`, passing `x` on, a slice) compiles as before with the parameter an opaque object. Other dtypes, including bool, are opaque objects. Results follow the interpreter: a value derived from an element is a numpy scalar there, so it is boxed as `np.float64` / `np.int64` / `np.bool` when it escapes (return, `Box`), decided per value at run time by a taint flag; a function declared `-> float` / `int` / `bool` that holds a numpy array returns the interpreter's object. `float(v)` of such a value is a plain float. An out-of-range index raises numpy's `IndexError("index i is out of bounds for axis 0 with size n")`. The numpy RuntimeWarnings (overflow, invalid value) and numpy's division are not emulated in this step: a pure function deopts on a non-finite float `+ - *` result or an unproved int overflow, so the interpreter emits them; `/`, `//`, `%` on a value derived from an element, and `+ - *` on one in an impure function, are not compiled | anything else: object path |
| A function holding an array parameter runs no user code: closed, and no slot read or `C(args)` whose slow path runs user code, in its whole call tree (#203) | not compiled; diagnostic |
| OBJ references follow one ownership rule; no IR form for free, borrowed alias or NULL read | not compiled; diagnostic |
| Integer overflow checked and promoted (deopt); an op marked `proven` must re-prove its interval fits i64 | not compiled; diagnostic `verify/proven-overflow` |
Expand Down
38 changes: 27 additions & 11 deletions python-multiplatform-ksp/src/main/python/typedpython/cgen.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,11 @@
(tp_runtime.h).

Buffer arrays (issue #149, identity-safe part). A parameter annotated `array.array[float]`,
`array.array[int]` is an `ArrayParam` with `buffer` "array". numpy: not supported yet (#220):
a numpy element's arithmetic produces np.float64 / np.int64 / np.bool (compiled code returns float),
numpy scalar operations emit observable RuntimeWarnings and divide by zero without raising (compiled
double arithmetic does neither), and emulating that needs taint tracking of numpy-derived values,
numpy-scalar boxing and FP-exception-to-warning emulation. An `NDArray[...]` parameter is an ordinary
object parameter. `tp_wrap_f` does not copy it: after the scalar guards, `tp_buf_enter`
`array.array[int]` is an `ArrayParam` with `buffer` "array"; `NDArray[np.float64]` / `NDArray[np.int64]`
(#220) is one with `buffer` "numpy" (exact `numpy.ndarray`, an out-of-range index raises numpy's
IndexError text). A numpy element is an np.float64 / np.int64 in the interpreter: `Box.np` boxes
the matching numpy scalar when its run-time flag is true, and `BinOp.np` deopts a non-finite float
`+ - *` result so the interpreter emits numpy's RuntimeWarning (taint.py). `tp_wrap_f` does not copy it: after the scalar guards, `tp_buf_enter`
checks the exact type (captured in `tp_state.buf_types` by the exec slot), takes one `Py_buffer`
export (C-contiguous, 1-D, 8-byte items, format `d` / `q` / `l`, aligned, writable when the function
stores) and any mismatch deopts before an effect. The export is released by `tp_buf_exit`, part of the
Expand Down Expand Up @@ -186,7 +185,9 @@ class CGenError(Exception):
_ARRAY_CT = {Type.F64_ARRAY: "tp_f64_array", Type.I64_ARRAY: "tp_i64_array"}
# Buffer arrays (#149): `ArrayParam.buffer` -> the exact type the guard compares against (the index
# into the module state's `buf_types`), and the format strings the export must carry.
_BUFFER_KINDS = {None: None, "array": 0}
_BUFFER_KINDS = {None: None, "array": 0, "numpy": 1}
# buf_types slots 2..4: the numpy scalar types a Box.np makes (np.float64, np.int64, np.bool_)
_NP_SLOT = {Type.F64: 2, Type.I64: 3, Type.BOOL: 4}
_BUFFER_FMT = {Type.F64_ARRAY: ('"d"', "NULL"), Type.I64_ARRAY: ('"q"', '"l"')}
_BUFFER_SFX = {Type.F64_ARRAY: "f64", Type.I64_ARRAY: "i64"}
_CT_ARRAY_REV = {"tp_f64_array": Type.F64_ARRAY, "tp_i64_array": Type.I64_ARRAY}
Expand Down Expand Up @@ -672,7 +673,7 @@ def generate(module: ir.Module, source_path: Path, display_path: str | None = No
w(f" PyObject *ints[{len(mg.int_consts) + 1}]; /* strong: the int constants boxed once (SPEC N-13) */")
w(f" tp_gslot gc[{len(mg.gnames) + 1}]; /* cached global/builtin reads (SPEC N-13) */")
w(" PyObject *builtins; /* strong: the builtins dict the module source ran with; NULL: none */")
w(" PyObject *buf_types[1]; /* strong: array.array (buffer arrays, #149); NULL: not importable */")
w(" PyObject *buf_types[5]; /* strong: array.array, numpy.ndarray, np.float64, np.int64, np.bool_ (buffer arrays #149, #220); NULL: not importable */")
w(" PyObject *mtype; /* strong: the descriptor type of compiled methods; NULL: none */")
w(f" PyObject *mdescrs[{mg.n_methods + 1}]; /* strong: the installed descriptor of each method */")
w("} tp_state;")
Expand Down Expand Up @@ -1048,6 +1049,9 @@ def ev_BinOp(self, e: ir.BinOp) -> str:
t = self.tmp("double")
if e.op in _F64_INLINE:
self.emit(f"{t} = {a} {_F64_INLINE[e.op]} {b};")
if e.np:
# #220: numpy would warn (overflow / invalid value); the interpreter redoes the call
self.emit(f"if (!isfinite({t})) {{ tp_rc = 1; goto tp_exit; }}")
else:
self.check(f"{_F64_OP[e.op]}({a}, {b}, &{t})")
return t
Expand Down Expand Up @@ -1155,6 +1159,8 @@ def ev_Truth(self, e: ir.Truth) -> str:
return t

def ev_ToFloat(self, e: ir.ToFloat) -> str:
if e.explicit and e.operand.type is Type.F64:
return self.ev(e.operand)
if e.operand.type is not Type.I64:
raise CGenError(f"{self.f.name}: ToFloat of {e.operand.type}")
v = self.ev(e.operand)
Expand Down Expand Up @@ -1194,6 +1200,12 @@ def ev_Box(self, e: ir.Box) -> str:
if ot not in _BOX:
raise CGenError(f"{self.f.name}: Box of {ot}")
v = self.ev(e.operand)
if e.np is not None:
if e.np.type is not Type.BOOL:
raise CGenError(f"{self.f.name}: Box np flag of {e.np.type}")
flag = self.ev(e.np)
return self._own_new(f"({flag} ? tp_box_np(tp_st->buf_types[{_NP_SLOT[ot]}], "
f"{_BOX[ot]}({v})) : {_BOX[ot]}({v}))")
return self._own_new(f"{_BOX[ot]}({v})")

def ev_Unbox(self, e: ir.Unbox) -> str:
Expand Down Expand Up @@ -2108,7 +2120,8 @@ def wrapper(self) -> str:
for i, p in buffers:
fmt1, fmt2 = _BUFFER_FMT[p.type]
kind = _BUFFER_KINDS[p.buffer]
L.append(f" tp_s = tp_buf_enter(tp_args[{i}], "
enter = "tp_buf_enter_np" if p.buffer == "numpy" else "tp_buf_enter"
L.append(f" tp_s = {enter}(tp_args[{i}], "
f"((tp_state *)PyModule_GetState(tp_module))->buf_types[{kind}], "
f"{fmt1}, {fmt2}, {int(p.stored)}, &tp_a{i});")
L.append(" if (tp_s < 0) goto tp_error;")
Expand Down Expand Up @@ -2403,7 +2416,7 @@ def _state_functions(mg: _ModGen) -> str:
" for (i = 0; i < sizeof(tp_st->ints) / sizeof(tp_st->ints[0]); i++) Py_VISIT(tp_st->ints[i]);",
" Py_VISIT(tp_st->mtype);",
" for (i = 0; i < sizeof(tp_st->mdescrs) / sizeof(tp_st->mdescrs[0]); i++) Py_VISIT(tp_st->mdescrs[i]);",
" for (i = 0; i < 1; i++) Py_VISIT(tp_st->buf_types[i]);",
" for (i = 0; i < sizeof(tp_st->buf_types) / sizeof(tp_st->buf_types[0]); i++) Py_VISIT(tp_st->buf_types[i]);",
" for (i = 0; i < sizeof(tp_st->names) / sizeof(tp_st->names[0]); i++) Py_VISIT(tp_st->names[i]);",
" for (i = 0; i < sizeof(tp_st->kwnames) / sizeof(tp_st->kwnames[0]); i++) Py_VISIT(tp_st->kwnames[i]);",
" for (i = 0; i < sizeof(tp_st->classes) / sizeof(tp_st->classes[0]); i++) Py_VISIT(tp_st->classes[i]);",
Expand All @@ -2425,7 +2438,7 @@ def _state_functions(mg: _ModGen) -> str:
" for (i = 0; i < sizeof(tp_st->gc) / sizeof(tp_st->gc[0]); i++) tp_gcache_clear(&tp_st->gc[i]);",
" for (i = 0; i < sizeof(tp_st->mdescrs) / sizeof(tp_st->mdescrs[0]); i++) tp_release(&tp_st->mdescrs[i]);",
" tp_release(&tp_st->mtype);",
" for (i = 0; i < 1; i++) tp_release(&tp_st->buf_types[i]);",
" for (i = 0; i < sizeof(tp_st->buf_types) / sizeof(tp_st->buf_types[0]); i++) tp_release(&tp_st->buf_types[i]);",
" for (i = 0; i < sizeof(tp_st->names) / sizeof(tp_st->names[0]); i++) tp_release(&tp_st->names[i]);",
" for (i = 0; i < sizeof(tp_st->kwnames) / sizeof(tp_st->kwnames[0]); i++) tp_release(&tp_st->kwnames[i]);",
" for (i = 0; i < sizeof(tp_st->cls_ok) / sizeof(tp_st->cls_ok[0]); i++) tp_st->cls_ok[i] = 0;",
Expand Down Expand Up @@ -2541,6 +2554,9 @@ def _exec_function(mg: _ModGen) -> str:
# module source ran (so its own imports are done). Not importable: NULL, and every guard deopts.
if "array" in kinds:
L.append(' if (tp_import_attr("array", "array", &tp_st->buf_types[0]) < 0) goto tp_done;')
if "numpy" in kinds:
for slot, attr in ((1, "ndarray"), (2, "float64"), (3, "int64"), (4, "bool_")):
L.append(f' if (tp_import_attr("numpy", "{attr}", &tp_st->buf_types[{slot}]) < 0) goto tp_done;')
L += [
" tp_table = PyDict_New();",
" if (tp_table == NULL || PyDict_SetItemString(tp_dict, \"__typedpython_interpreted__\", tp_table) < 0) goto tp_done;",
Expand Down
Loading