TypedPython: memoryview buffer parameters, BufferError on resize (#149)

benchmarks/typedpython/py/buffer_calls.py

@@ -0,0 +1,52 @@
+import array
+import sys
+
+
+def bump_list(a: list[float], i: int) -> None:
+    a[i] += 1.0
+
+
+def bump_array(a: array.array[float], i: int) -> None:
+    a[i] += 1.0
+
+
+def bump_view(a: memoryview[float], i: int) -> None:
+    a[i] += 1.0
+
+
+def run_list(a: list[float], calls: int) -> None:
+    n: int = len(a)
+    for k in range(calls):
+        bump_list(a, (k * 7919) % n)
+
+
+def run_array(a: array.array[float], calls: int) -> None:
+    n: int = len(a)
+    for k in range(calls):
+        bump_array(a, (k * 7919) % n)
+
+
+def run_view(a: memoryview[float], calls: int) -> None:
+    n: int = len(a)
+    for k in range(calls):
+        bump_view(a, (k * 7919) % n)
+
+
+def main() -> None:
+    # argv[1] = number of bump calls, argv[2] = array size (default 1000000). Each call touches one element
+    # of a large buffer: a list argument is copied in and written back on every call (O(n)), an exact
+    # array.array / memoryview argument is used in place (SPEC N-8, #149). Output: one checksum per variant.
+    calls: int = int(sys.argv[1])
+    size: int = int(sys.argv[2]) if len(sys.argv) > 2 else 1000000
+    as_list: list[float] = [0.0] * size
+    as_array: array.array[float] = array.array("d", [0.0]) * size
+    backing: array.array[float] = array.array("d", [0.0]) * size
+    as_view: memoryview[float] = memoryview(backing)
+    run_list(as_list, calls)
+    run_array(as_array, calls)
+    run_view(as_view, calls)
+    print(f"{sum(as_list)} {sum(as_array)} {sum(backing)}")
+
+
+if __name__ == "__main__":
+    main()

docs/SPEC.md

@@ -688,7 +688,7 @@ All in `PM/commonTest`, so they run wherever the interpreter loads.
   | Array access in bounds, or a runtime bounds check kept | the check stays (never an unchecked access) |
   | Array parameters only indexed, stored, `len`-ed; no aliasing (entry guard) | not compiled; diagnostic |
   | Integer ops proven safe by the verifier's intervals (constants, `len`, `range` variables, guards such as `if x < e` / `while lo < hi`, joined local assignments, `//` and `%` by constants) | emitted unchecked (a shift/mask for `//`/`%` by a power of two of a non-negative value); the sanitizer CI run builds with `TP_CHECK_PROVEN` and turns a wrong proof into a SystemError (#148) |
-  | `array.array[float]` / `array.array[int]` parameters (exact `array.array`, typecode `d` or `q`/`l`, 8-byte items) are read and written in place: the buffer export is held for the call and released on every exit, and every element access is one relaxed atomic 8-byte load or store, so each access is atomic like the interpreter's `a[i]`; no copy-in and no limit on the number of buffers. Open (#149): while the export is held, another thread's resize of that array raises BufferError. numpy arrays: see the next row | anything else: object path |
+  | `array.array[float]` / `array.array[int]` parameters (exact `array.array`, typecode `d` or `q`/`l`, 8-byte items) are read and written in place: the buffer export is held for the call and released on every exit, and every element access is one relaxed atomic 8-byte load or store, so each access is atomic like the interpreter's `a[i]`; no copy-in and no limit on the number of buffers. While the export is held, a resize of that array (`append`, `extend`, `del a[i]`, ... from another thread) raises BufferError, exactly as CPython does for any held export; the compiled call never sees a resize. Concurrent writes from another thread to the same elements are not synchronised: as in CPython, no ordering between the two threads' element accesses is guaranteed, only that each access is whole (no torn element). `memoryview[float]` / `memoryview[int]` parameters (#149: exact `memoryview`, 1-D, C-contiguous, format `d` or `q`/`l`, 8-byte aligned, writable when stored to; any exporter: an `array.array`, a numpy array, a `bytearray` cast with `.cast('d')`) take the same path with array.array's element semantics (an element is a Python float / int); the export is taken on the memoryview, so while the call runs `release()` on it raises BufferError, as CPython does for an exported view. An out-of-range index raises memoryview's `IndexError("index out of bounds on dimension 1")`. A released, read-only (when stored to), non-contiguous, multi-dimensional or other-format view deopts: the interpreter raises or runs it. numpy arrays: see the next row | anything else: object path |
   | numpy `npt.NDArray[np.float64]` / `NDArray[np.int64]` parameters (exact `numpy.ndarray`, 1-D, C-contiguous, native byte order, writable when stored to) take the same buffer path (#220, step 1) when the function accesses the array purely by element (`x[i]`, `len(x)`); a function that uses it as a whole object (`np.sum(x)`, `x.T`, passing `x` on, a slice) compiles as before with the parameter an opaque object. `NDArray[np.bool_]` (#261) is the same path with 1-byte items (buffer format `?`; a `uint8` or other array deopts): an element read is a native bool, boxed as `np.bool` when it escapes; a store compiles only for a bool value (`x[i] = v`, `v: bool`), because numpy converts an int or float by truthiness and that function stays interpreted. Other dtypes are opaque objects. Results follow the interpreter: a value derived from an element is a numpy scalar there, so it is boxed as `np.float64` / `np.int64` / `np.bool` when it escapes (return, `Box`), decided per value at run time by a taint flag; a function declared `-> float` / `int` / `bool` that holds a numpy array returns the interpreter's object. `float(v)` of such a value is a plain float. An out-of-range index raises numpy's `IndexError("index i is out of bounds for axis 0 with size n")`. Arithmetic on a value derived from an element follows numpy's scalar semantics (#220, step 2): float `/`, `//`, `%` never raise ZeroDivisionError (inf / nan, with numpy's result bits and sign rules, `//` and `%` as numpy's `divmod`), int64 `//` and `%` by zero give 0, int64 `/` converts both operands to double. numpy's RuntimeWarning (category and text exactly as numpy's, e.g. `overflow encountered in scalar add`, `divide by zero encountered in scalar divide`, `invalid value encountered in scalar floor_divide`) is emitted once per flagged operation, with the source file and line of the operation. A pure function does not emit it itself: on any floating-point exception (overflow, invalid, divide by zero) or int64 wrap or division by zero it deopts, so the interpreter emits it; an impure function cannot deopt after an effect, so the compiled code emits the warning (`PyErr_WarnExplicitObject`, honouring `warnings` filters and the module's `__warningregistry__`). Float `+ - * / // %` on such values is compiled in both kinds of function, including a store of the result into the array (`x[i] -= d * m`); int64 `/ // %` is compiled in a pure function only. Int64 `+ - *` and `// %` of an element in an impure function (a native INT would have to be a numpy scalar) stay on the object path | anything else: object path |
   | A function holding an array parameter runs no user code: closed, and no slot read or `C(args)` whose slow path runs user code, in its whole call tree (#203) | not compiled; diagnostic |
   | OBJ references follow one ownership rule; no IR form for free, borrowed alias or NULL read | not compiled; diagnostic |

python-multiplatform-ksp/src/main/python/typedpython/cgen.py

@@ -186,7 +186,7 @@ class CGenError(Exception):
 _ARRAY_CT = {Type.F64_ARRAY: "tp_f64_array", Type.I64_ARRAY: "tp_i64_array"}
 # Buffer arrays (#149): `ArrayParam.buffer` -> the exact type the guard compares against (the index
 # into the module state's `buf_types`), and the format strings the export must carry.
-_BUFFER_KINDS = {None: None, "array": 0, "numpy": 1}
+_BUFFER_KINDS = {None: None, "array": 0, "numpy": 1, "memoryview": None}
 # buf_types slots 2..4: the numpy scalar types a Box.np makes (np.float64, np.int64, np.bool_)
 _NP_SLOT = {Type.F64: 2, Type.I64: 3, Type.BOOL: 4}
 _BUFFER_FMT = {Type.F64_ARRAY: ('"d"', "NULL"), Type.I64_ARRAY: ('"q"', '"l"'), Type.BOOL_ARRAY: ('"?"', "NULL")}
@@ -2784,9 +2784,10 @@ def wrapper(self) -> str:
         for i, p in buffers:
             fmt1, fmt2 = _BUFFER_FMT[p.type]
             kind = _BUFFER_KINDS[p.buffer]
-            enter = "tp_buf_enter_np" if p.buffer == "numpy" else "tp_buf_enter"
-            L.append(f"    tp_s = {enter}(tp_args[{i}], "
-                     f"((tp_state *)PyModule_GetState(tp_module))->buf_types[{kind}], "
+            enter = {"numpy": "tp_buf_enter_np", "memoryview": "tp_buf_enter_mv"}.get(p.buffer, "tp_buf_enter")
+            exact = ("(PyObject *)&PyMemoryView_Type" if p.buffer == "memoryview"
+                     else f"((tp_state *)PyModule_GetState(tp_module))->buf_types[{kind}]")
+            L.append(f"    tp_s = {enter}(tp_args[{i}], {exact}, "
                      f"{fmt1}, {fmt2}, {int(p.stored)}, &tp_a{i});")
             L.append("    if (tp_s < 0) goto tp_error;")
             L.append("    if (tp_s != 0) goto tp_deopt;")

python-multiplatform-ksp/src/main/python/typedpython/frontend.py

@@ -326,7 +326,7 @@ def _qualified(node: ast.expr, info: _ModuleInfo) -> str | None:
 def _buffer_kind(annotation: ast.expr, info: _ModuleInfo) -> tuple[Type, str] | None:
     """The buffer-array reading of an annotation (#149), else None (an ordinary object parameter):
     `array.array[float]` / `array.array[int]` (any spelling that resolves, through this module's
-    own imports, to `array.array`). A bare `array.array`, another element type, a numpy
+    own imports, to `array.array`), and the builtin `memoryview[float]` / `memoryview[int]`. A bare `array.array`, another element type, a numpy
     `NDArray[...]` (not supported: its element arithmetic is numpy's), or a name bound more than
     once is not one."""
     if not isinstance(annotation, ast.Subscript):
@@ -336,6 +336,10 @@ def _buffer_kind(annotation: ast.expr, info: _ModuleInfo) -> tuple[Type, str] |
     if base == "array.array" and isinstance(arg, ast.Name) and arg.id in ("float", "int") \
             and not info.bound(arg.id):
         return ARRAY_KIND[f"list[{arg.id}]"], "array"
+    if isinstance(annotation.value, ast.Name) and annotation.value.id == "memoryview" \
+            and not info.bound("memoryview") and isinstance(arg, ast.Name) \
+            and arg.id in ("float", "int") and not info.bound(arg.id):
+        return ARRAY_KIND[f"list[{arg.id}]"], "memoryview"
     if base == "numpy.typing.NDArray":
         dtype = _qualified(arg, info)
         if dtype in _NUMPY_DTYPE:

python-multiplatform-ksp/src/main/python/typedpython/runtime/API.md

@@ -95,8 +95,9 @@ An exact `array.array` argument (`array.array[float]` / `array.array[int]`; the
 through the module's own top-level imports) is not copied.
     typedef struct { Py_buffer view; Py_ssize_t len; void *data; int held; } tp_buf;
     int  tp_buf_enter(PyObject *obj, PyObject *exact, const char *fmt1, const char *fmt2, int writable, tp_buf *b);
+    int  tp_buf_enter_mv(PyObject *obj, PyObject *exact, const char *fmt1, const char *fmt2, int writable, tp_buf *b);   /* memoryview (#149): memoryview's IndexError text */
     void tp_buf_exit(tp_buf *b);          /* PyBuffer_Release if held; zeroes b; idempotent */
-    int  tp_buf_slot(const tp_buf *b, int64_t i, int store, Py_ssize_t *slot);   /* 0 / -1, array.array's IndexError text */
+    int  tp_buf_slot(const tp_buf *b, int64_t i, int store, Py_ssize_t *slot);   /* 0 / -1, array.array's, numpy's or memoryview's IndexError text */
     double  tp_buf_ld_f64(const tp_buf *b, Py_ssize_t i);   void tp_buf_st_f64(const tp_buf *b, Py_ssize_t i, double v);
     int64_t tp_buf_ld_i64(const tp_buf *b, Py_ssize_t i);   void tp_buf_st_i64(const tp_buf *b, Py_ssize_t i, int64_t v);
     int  tp_import_attr(const char *mod, const char *attr, PyObject **out);   /* exec slot: new ref or NULL (cleared); -1 MemoryError */

python-multiplatform-ksp/src/main/python/typedpython/runtime/tp_runtime.h

@@ -1224,8 +1224,22 @@ static inline int tp_buf_enter_np(PyObject *obj, PyObject *exact, const char *fm
     return rc;
 }
 
+/* A memoryview (#149): the same guard on the exact builtin memoryview (the export is taken on the
+ * view itself, so its release() raises BufferError while the call runs); `np` 2 selects
+ * memoryview's IndexError text. Elements are Python floats / ints, as for array.array. */
+static inline int tp_buf_enter_mv(PyObject *obj, PyObject *exact, const char *fmt1, const char *fmt2,
+                                  int writable, tp_buf *b)
+{
+    int rc = tp_buf_enter(obj, exact, fmt1, fmt2, writable, b);
+    if (rc == 0) {
+        b->np = 2;
+    }
+    return rc;
+}
+
 /* index: a negative index counts from the end; out of range -> IndexError with the text CPython's
- * own array.array uses ("array index out of range" / "array assignment index out of range") */
+ * own array.array uses ("array index out of range" / "array assignment index out of range"),
+ * numpy's (np 1) or memoryview's (np 2) */
 static inline int tp_buf_slot(const tp_buf *b, int64_t i, int store, Py_ssize_t *slot)
 {
     int64_t orig = i;
@@ -1239,6 +1253,10 @@ static inline int tp_buf_slot(const tp_buf *b, int64_t i, int store, Py_ssize_t
     *slot = (Py_ssize_t)i;
     return 0;
 out_of_range:
+    if (b->np == 2) {
+        PyErr_SetString(PyExc_IndexError, "index out of bounds on dimension 1");
+        return -1;
+    }
     if (b->np) {
         PyErr_Format(PyExc_IndexError, "index %lld is out of bounds for axis 0 with size %zd",
                      (long long)orig, b->len);

python-multiplatform-ksp/src/main/python/typedpython/verify.py

@@ -45,7 +45,7 @@
   verify/redo                 `Call.redo` where it is not legal (see below)
   verify/entry-global-open    a function with entry_globals is not closed
   verify/array-open           a function with an ArrayParam runs user code (#203)
-  verify/buffer               a malformed `ArrayParam.buffer` (only None and "array"), or a
+  verify/buffer               a malformed `ArrayParam.buffer` (None, "array", "numpy", "memoryview"), or a
                               CopyArray of a buffer array (#149)
   verify/par-loop             a `ForRange.par` (SPEC N-15, #277) that the verifier does not recompute to
                               the same value: the function is not marked `parallel`, or the loop is
@@ -450,7 +450,7 @@ def declarations(self) -> frozenset[str]:
                     self.error(TYPE, f"array parameter '{p.name}' has non-array type {p.type}")
                 elif not isinstance(p.stored, bool):
                     self.error(TYPE, f"array parameter '{p.name}' has a non-bool `stored`")
-                elif p.buffer not in (None, "array", "numpy"):
+                elif p.buffer not in (None, "array", "numpy", "memoryview"):
                     self.error(BUFFER, f"array parameter '{p.name}' has buffer kind {p.buffer!r}")
                 else:
                     self.arrays[p.name] = p.type

python-multiplatform-ksp/src/test/python/test_buffer_arrays.py

@@ -472,7 +472,7 @@ def test_verifier_buffer_kind_and_copy_rules():
     from test_verify import F64 as VF64, F64A as VF64A, assert_rejected, fn
     from typedpython.ir import ArrayParam, Assign, Const, CopyArray, Index, Return, Type
 
-    for kind in ("ndarray", "memoryview"):        # "numpy" is a buffer kind since #220
+    for kind in ("ndarray", "bytes"):        # "numpy" (#220) and "memoryview" (#149) are buffer kinds
         assert_rejected("verify/buffer", fn("f", params=[ArrayParam("a", VF64A, False, kind)],
                                              returns=VF64, pure=True,
                                              body=[Return(Index(VF64, "a", Const(Type.I64, 0)))]))
@@ -483,6 +483,8 @@ def test_verifier_buffer_kind_and_copy_rules():
     import dataclasses
     assert_rejected("verify/buffer", dataclasses.replace(
         copy, params=[ArrayParam("a", VF64A, False, "numpy")]))
+    assert_rejected("verify/buffer", dataclasses.replace(
+        copy, params=[ArrayParam("a", VF64A, False, "memoryview")]))
 
 
 def test_verifier_buffer_params_are_not_limited_but_lists_still_are():