diff --git a/docs/SPEC.md b/docs/SPEC.md index a26ad109..3f09c746 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -676,7 +676,7 @@ All in `PM/commonTest`, so they run wherever the interpreter loads. `Status: implemented` for desktop host builds (`test_pipeline.py`, `test_cgen*.py`, `test_recursion.py`, `test_eval_breaker.py`, `test_fast_entry.py`, `test_free_threaded.py`; free-threaded 3.15t, #159). Not yet: shipping compiled modules in an app through - pypackpack (#25), methods of fixed-layout classes (#155), native ints in impure functions (#232). + pypackpack (#25), methods of fixed-layout classes (#155). Native ints in impure functions are N-13 (#232). - **N-9** TypedPython memory safety is proved on the IR before C is generated (#41; maintainer decision 2026-10-03). Each property, and what happens when it cannot be proved: @@ -771,6 +771,68 @@ All in `PM/commonTest`, so they run wherever the interpreter loads. implementation ("An implementation is allowed to postpone garbage collection or omit it altogether", Data model §3.1), and CPython itself changed it in 3.14 (incremental GC). Everything else above stays exact. `Status: planned (design)`. +- **N-13** TypedPython native ints in impure functions (#232; INTENT §1.4: compiled code computes what + CPython computes). An impure function cannot deopt after an effect, so its `int` locals were CPython + objects and every `+`, `//`, `%` and comparison was a generic `PyNumber_*` / `PyObject_RichCompare` + call, slower than CPython 3.15's specialising interpreter. The IR has a type `INT` for them: a Python + int that is **either a native i64 or, after an overflow, a PyLong carried for the rest of that value's + life**. A function never deopts because of an `INT`; it promotes in place. + - **Where an `INT` is.** An `int`-typed local of an impure function (not a `range` target, not a + parameter) whose every assignment lowers to an `INT` (an arithmetic result on `INT`/i64 operands, an + int literal, a native i64 value widened by `ToInt`, or the result of a redo `Call`, below). A local + that is also assigned from an object (a call of an unknown callable, a global, `int(x)`) stays an + object, as before. An `INT` is created only inside a function; parameters, returns and entry globals + stay i64 or object, so the signature and the entry guards are unchanged. + - **Arithmetic.** `+ - * // % << >>` and unary `-` and `+` on `INT` operands run natively when both + operands are native and the result fits i64. When an operand is a PyLong, or the native result does + not fit (`2**63-1 + 1`, `-2**63 - 1`, `2**62 * 4`, `-(-2**63)`, `-2**63 // -1`, a left shift past + bit 63), the operation is CPython's own `PyNumber_*` on the two values as exact ints, and the result + is native again if it fits i64, else a PyLong. The result is bit-for-bit what the interpreter + computes, including after side effects; nothing is redone and nothing deopts. `//` and `%` use + floor semantics (sign of the divisor), a zero divisor raises CPython's `ZeroDivisionError` with the + running interpreter's message, a negative shift count raises `ValueError('negative shift count')`, + a shift by a huge count raises CPython's `OverflowError` / `MemoryError`. `/` of ints is not an + `INT` operation: both operands are boxed and `PyNumber_TrueDivide` runs (the result is a float + object). + - **Comparison.** `==`, `!=`, `<`, `<=`, `>`, `>=` of two `INT`s is a native compare when both are + native, else the exact comparison of the two ints; the result is a bool. `if x:` on an `INT` is + `x != 0`. + - **Boxing.** An `INT` becomes an object only at an object boundary (an argument of an object call, + a store into a container, an f-string field, a return value, an operand of an object operation): + a native value is boxed with `PyLong_FromLongLong`, a promoted one is the PyLong it already is. A + constant used as an object is built once, at module exec, and read from module state (a new + reference per use), not boxed per use. + - **Calls.** A direct call from an impure function of a pure compiled callee that may deopt and + returns i64 is a redo call (SPEC N-8, `Call.redo`): the callee's deopt, including an argument that is + a promoted `INT` (it does not fit the callee's i64 parameter), is absorbed by running the + interpreted callee on the boxed arguments (unobservable, the callee is pure). The result is an `INT`: + exactly an `int` object (a native or a promoted value); any other object raises `SystemError` + (`typedpython: the interpreted redo of f did not return an int`), as the existing redo does. + - **Globals and builtins.** A module global or builtin read in an impure function (`Global`) is cached + per name in module state and revalidated by the dict-watcher epoch (N-11): the cached object is used + while no watched key of the module dict (or of the builtins dict) has changed, and any change to a + cached name, a clear, a rebinding of `__builtins__`, or a failed watcher registration sends every + read back to the full lookup (`tp_global`). The cache holds a borrowed pointer and takes its own + reference on each hit, so it extends no lifetime: a rebound function, type or module is freed, and + the `__del__` of anything it holds runs, exactly when CPython would run them (free-threaded builds + take the reference with `PyUnstable_TryIncRef`, CPython 3.14+). The result is what LOAD_GLOBAL would + return at that moment; a name that is unbound still raises `NameError`. This also removes the + per-call creation of the `"__builtins__"` string. + - **Cost rule (#150).** An `INT` operation (`BinOp`, `Compare`, `UnaryOp` on `INT` operands, a `Call` + whose arguments are all native or `INT`) counts as native work, so an impure function whose loops do + arithmetic on `INT`s is compiled. This includes such an operation directly in the body of a loop over an + object (`for x in obj`, `for _ in range(n)` with an object `n`), which the rule otherwise ignores; float + and array work in an object loop still does not count. A loop that does only object operations is + still rejected. + wordfreq's `word_of` and `main` are compiled. + - **Proof.** The verifier checks the types of every `INT` node (an `INT` operation takes `INT` + operands and gives an `INT`; `proven` is never set on one; an `INT` flows into an object slot only + through `Box`; a `INT` local is declared and definitely assigned like any other) and the call rule + above. The generated C releases an `INT`'s PyLong on every exit, as for any owned reference. + - **Tests.** `test_native_int_impure.py`: i64 locals in impure functions; the i64 boundary for add, + sub, mul, neg, floordiv, mod and shifts, after side effects and with no deopt, each equal to the + interpreter; constants pre-boxed; global and builtin reads cached and invalidated by rebinding; + wordfreq's `word_of` and `main` compile and print what CPython prints. `Status: implemented`. --- diff --git a/python-multiplatform-ksp/src/main/python/typedpython/cgen.py b/python-multiplatform-ksp/src/main/python/typedpython/cgen.py index 0cab9493..7b1ea1bf 100644 --- a/python-multiplatform-ksp/src/main/python/typedpython/cgen.py +++ b/python-multiplatform-ksp/src/main/python/typedpython/cgen.py @@ -181,7 +181,8 @@ class CGenError(Exception): """The IR cannot be lowered as specified (a contract violation by the producer of the IR).""" -_CT = {Type.I64: "int64_t", Type.F64: "double", Type.BOOL: "int", Type.OBJ: "PyObject *"} +_CT = {Type.I64: "int64_t", Type.F64: "double", Type.BOOL: "int", Type.OBJ: "PyObject *", + Type.INT: "tp_int"} _ARRAY_CT = {Type.F64_ARRAY: "tp_f64_array", Type.I64_ARRAY: "tp_i64_array"} # Buffer arrays (#149): `ArrayParam.buffer` -> the exact type the guard compares against (the index # into the module state's `buf_types`), and the format strings the export must carry. @@ -202,6 +203,11 @@ class CGenError(Exception): ir.BinOpKind.MOD: "tp_mod_f64"} _F64_INLINE = {ir.BinOpKind.ADD: "+", ir.BinOpKind.SUB: "-", ir.BinOpKind.MUL: "*"} _OBJ_OP = {k: f"TP_BINOP_{k.name}" for k in ir.BinOpKind} +# SPEC N-13: the INT operations (tp_runtime.h "Native ints"); TRUEDIV of ints is an OBJ operation +_INT_OP = {ir.BinOpKind.ADD: "tp_int_add", ir.BinOpKind.SUB: "tp_int_sub", + ir.BinOpKind.MUL: "tp_int_mul", ir.BinOpKind.FLOORDIV: "tp_int_floordiv", + ir.BinOpKind.MOD: "tp_int_mod", ir.BinOpKind.LSHIFT: "tp_int_lshift", + ir.BinOpKind.RSHIFT: "tp_int_rshift"} _MATH = {ir.MathFunc.SQRT: 1, ir.MathFunc.EXP: 1, ir.MathFunc.LOG: 1, ir.MathFunc.SIN: 1, ir.MathFunc.COS: 1, ir.MathFunc.TAN: 1, ir.MathFunc.FABS: 1, ir.MathFunc.ATAN2: 2, ir.MathFunc.HYPOT: 2} @@ -395,7 +401,7 @@ def _class_current_expr(obj_c: str, k: int, fields_c: str, n: int, descrs_c: str def _watch_event_stmt() -> str: """The body of the module's dict-watcher callback (module docstring, rule 5).""" - return "return tp_globals_watch_event(event, key, tp_class_names);" + return "return tp_globals_watch_event(event, key, tp_watch_names);" def _is_effect(e: ir.Expr) -> bool: @@ -424,7 +430,7 @@ def _runs_no_python(f: ir.Function) -> bool: return False if any(isinstance(p, ir.Param) and p.type is Type.OBJ for p in f.params): return False - if any(t is Type.OBJ for t in f.locals.values()): + if any(t in (Type.OBJ, Type.INT) for t in f.locals.values()): return False for s in _walk_stmts(f.body): if isinstance(s, (ir.FieldSet, ir.StoreItem, ir.ForObj)): @@ -432,7 +438,8 @@ def _runs_no_python(f: ir.Function) -> bool: for e in _all_exprs(f): if isinstance(e, _RUNS_PYTHON): return False - if e.type is Type.OBJ and not (isinstance(e, ir.Box) and e.operand.type in ir.SCALARS): + if e.type is Type.INT or (e.type is Type.OBJ and not (isinstance(e, ir.Box) + and e.operand.type in ir.SCALARS)): return False return True @@ -512,6 +519,8 @@ def __init__(self, module: ir.Module): self.method_base[cd.name] = total_methods total_methods += len(self.methods_of[cd.name]) self.n_methods = total_methods + self.int_consts: list[int] = [] # module-state boxed int constants (SPEC N-13) + self.gnames: list[str] = [] # cached global/builtin reads, one tp_gslot each self._uses: dict[str, set[str]] = {} self.bounded = _bounded_functions(self.functions) # name -> k (module docstring, "Fast entry") @@ -582,6 +591,20 @@ def watches_globals(self) -> bool: return any(isinstance(e, (ir.New, ir.IsExact)) for f in self.module.functions for e in _all_exprs(f)) + def needs_watch(self) -> bool: + """A dict watcher on the module dict (and the builtins dict): class guards or cached reads.""" + return self.watches_globals() or bool(self.gnames) + + def int_index(self, v: int) -> int: + if v not in self.int_consts: + self.int_consts.append(v) + return self.int_consts.index(v) + + def gslot_index(self, name: str) -> int: + if name not in self.gnames: + self.gnames.append(name) + return self.gnames.index(name) + def name_index(self, s: str) -> int: if s not in self.names: self.names.append(s) @@ -646,6 +669,9 @@ def generate(module: ir.Module, source_path: Path, display_path: str | None = No w(f" tp_class_rt cls_rt[{len(module.classes) + 1}]; /* version-tag / global guard state */") w(" int watch_id1; /* dict watcher id + 1 on the module dict; 0 = none */") w(" PyObject *breaker; /* `lambda: None`; calling it runs the eval breaker (#141) */") + w(f" PyObject *ints[{len(mg.int_consts) + 1}]; /* strong: the int constants boxed once (SPEC N-13) */") + w(f" tp_gslot gc[{len(mg.gnames) + 1}]; /* cached global/builtin reads (SPEC N-13) */") + w(" PyObject *builtins; /* strong: the builtins dict the module source ran with; NULL: none */") w(" PyObject *buf_types[1]; /* strong: array.array (buffer arrays, #149); NULL: not importable */") w(" PyObject *mtype; /* strong: the descriptor type of compiled methods; NULL: none */") w(f" PyObject *mdescrs[{mg.n_methods + 1}]; /* strong: the installed descriptor of each method */") @@ -665,11 +691,14 @@ def generate(module: ir.Module, source_path: Path, display_path: str | None = No w(f"static const char *const tp_mnames_{mg.class_index[cd.name]}[] = {{{items}NULL}};") names = "".join(f"{c_string(cd.name)}, " for cd in module.classes) w(f"static const char *const tp_class_names[] = {{{names}NULL}};") + watch = [cd.name for cd in module.classes] + list(mg.gnames) + (["__builtins__"] if mg.gnames else []) + w("static const char *const tp_watch_names[] = {" + "".join(f"{c_string(x)}, " for x in dict.fromkeys(watch)) + "NULL};") w("") if mg.n_methods: w(_TEMPLATE_METHOD_TYPE) - if module.classes and mg.watches_globals(): - w("/* Dict watcher of the module dict (tp_globals_watch): a class-name key changed. */") + if mg.needs_watch(): + w("/* Dict watcher of the module dict and its builtins dict (tp_globals_watch): a class name or a cached") + w(" global changed. */") w("static int tp_watch_cb(PyDict_WatchEvent event, PyObject *dict, PyObject *key, PyObject *value)") w("{") w(" (void)dict; (void)value;") @@ -751,9 +780,10 @@ def _check_function(f: ir.Function, functions: dict[str, ir.Function]) -> None: if any(isinstance(p, ir.ArrayParam) for p in callee.params): raise CGenError(f"{f.name}: Call of {e.function!r}, which takes array params") if e.redo: - if callee.returns not in (Type.F64, Type.BOOL, Type.NONE): + if callee.returns not in (Type.F64, Type.BOOL, Type.NONE) and not ( + callee.returns is Type.I64 and e.type is Type.INT): raise CGenError(f"{f.name}: redo Call of {e.function!r}, which returns " - f"{callee.returns} (ir.py: F64, BOOL or NONE only)") + f"{callee.returns} (ir.py: F64, BOOL or NONE, or I64 as an INT)") if not callee.pure: raise CGenError(f"{f.name}: redo Call of impure {e.function!r} " "(redoing it could repeat an effect)") @@ -774,7 +804,7 @@ def _check_function(f: ir.Function, functions: dict[str, ir.Function]) -> None: raise CGenError(f"{f.name}: array param {p.name} has buffer kind {p.buffer!r}") if p.stored and f.pure: raise CGenError(f"{f.name}: array param {p.name} is stored into in a pure function") - elif p.type not in _CT: + elif p.type not in _CT or p.type is Type.INT: raise CGenError(f"{f.name}: param {p.name} has type {p.type}") elif p.cls is not None and p.type is not Type.OBJ: raise CGenError(f"{f.name}: param {p.name} has a class guard but type {p.type}") @@ -935,6 +965,9 @@ def _own_release_all(self) -> list[str]: objs = [_ident("l", n) for n, t in self.types.items() if t is Type.OBJ] objs += [n for ct, n in self.temps if ct == "PyObject *"] out = [f" tp_release(&{n});" for n in objs] + ints = [_ident("l", n) for n, t in self.types.items() if t is Type.INT] + ints += [n for ct, n in self.temps if ct == "tp_int"] + out += [f" tp_int_release(&{n});" for n in ints] # local arrays and array temporaries: freed on every exit (ok, error, deopt); idempotent arrs = [(_ident("l", n), t) for n, t in self.local_arrays.items()] arrs += [(n, _CT_ARRAY_REV[ct]) for ct, n in self.temps if ct in _CT_ARRAY_REV] @@ -976,6 +1009,10 @@ def ev_Local(self, e: ir.Local) -> str: raise CGenError(f"{self.f.name}: local {e.name} is {t}, read as {e.type}") if t is Type.OBJ: return self._own_copy(_ident("l", e.name)) + if t is Type.INT: # an owned copy, like an OBJ local (the native case is two loads) + c = self.tmp("tp_int") + self.emit(f"tp_int_copy(&{c}, &{_ident('l', e.name)});") + return c return _ident("l", e.name) def ev_BinOp(self, e: ir.BinOp) -> str: @@ -983,6 +1020,16 @@ def ev_BinOp(self, e: ir.BinOp) -> str: if lt is not rt: raise CGenError(f"{self.f.name}: BinOp operands {lt} and {rt}") a, b = self.ev(e.left), self.ev(e.right) + if lt is Type.INT: + if e.op not in _INT_OP or e.proven or e.type is not Type.INT: + raise CGenError(f"{self.f.name}: BinOp {e.op.name} on INT operands") + t = self.tmp("tp_int") + self.emit(f"tp_s = {_INT_OP[e.op]}(&{a}, &{b}, &{t});") + self.emit(f"tp_int_release(&{a}); tp_int_release(&{b});") + self.emit("if (tp_s != 0) { tp_rc = -1; goto tp_exit; }") + return t + if lt in (Type.I64, Type.F64) and e.op in (ir.BinOpKind.LSHIFT, ir.BinOpKind.RSHIFT): + raise CGenError(f"{self.f.name}: {e.op.value} on {lt} operands (only INT and OBJ shift)") if lt is Type.I64: t = self.tmp("double" if e.op is ir.BinOpKind.TRUEDIV else "int64_t") if e.proven and e.op in _SHIFT_OPS: @@ -1016,6 +1063,16 @@ def ev_BinOp(self, e: ir.BinOp) -> str: def ev_UnaryOp(self, e: ir.UnaryOp) -> str: ot = e.operand.type v = self.ev(e.operand) + if ot is Type.INT and e.op in (ir.UnaryOpKind.NEG, ir.UnaryOpKind.POS): + if e.proven or e.type is not Type.INT: + raise CGenError(f"{self.f.name}: {e.op} of INT") + if e.op is ir.UnaryOpKind.POS: + return v # the operand's temporary is the result + t = self.tmp("tp_int") + self.emit(f"tp_s = tp_int_neg(&{v}, &{t});") + self.emit(f"tp_int_release(&{v});") + self.emit("if (tp_s != 0) { tp_rc = -1; goto tp_exit; }") + return t if e.op is ir.UnaryOpKind.NOT: if ot is not Type.BOOL: raise CGenError(f"{self.f.name}: NOT of {ot}") @@ -1043,7 +1100,11 @@ def ev_Compare(self, e: ir.Compare) -> str: lt, rt = e.left.type, e.right.type a, b = self.ev(e.left), self.ev(e.right) t = self.tmp("int") - if lt is rt and lt in ir.SCALARS: + if lt is Type.INT and rt is Type.INT: + self.emit(f"{t} = tp_int_cmp(&{a}, &{b}, {_PYCMP[e.op]});") + self.emit(f"tp_int_release(&{a}); tp_int_release(&{b});") + self.emit(f"if ({t} < 0) {{ tp_rc = -1; goto tp_exit; }}") + elif lt is rt and lt in ir.SCALARS: self.emit(f"{t} = ({a}) {e.op.value} ({b});") elif (lt, rt) == (Type.I64, Type.F64): self.emit(f"{t} = tp_cmp_i64_f64({a}, {b}, {_PYCMP[e.op]});") @@ -1101,6 +1162,14 @@ def ev_ToFloat(self, e: ir.ToFloat) -> str: self.emit(f"{t} = tp_i64_to_f64({v});") return t + def ev_ToInt(self, e: ir.ToInt) -> str: + if e.operand.type is not Type.I64 or e.type is not Type.INT: + raise CGenError(f"{self.f.name}: ToInt of {e.operand.type}") + v = self.ev(e.operand) + t = self.tmp("tp_int") + self.emit(f"{t}.v = {v}; {t}.big = NULL;") + return t + def ev_ObjToFloat(self, e: ir.ObjToFloat) -> str: if e.operand.type is not Type.OBJ: raise CGenError(f"{self.f.name}: ObjToFloat of {e.operand.type}") @@ -1113,6 +1182,15 @@ def ev_ObjToFloat(self, e: ir.ObjToFloat) -> str: def ev_Box(self, e: ir.Box) -> str: ot = e.operand.type + k = e.operand.operand if isinstance(e.operand, ir.ToInt) else e.operand + if isinstance(k, ir.Const) and k.type is Type.I64 and ot in (Type.I64, Type.INT): + # a constant is boxed once, at module exec (SPEC N-13), and read from module state + return self._own_copy(f"tp_st->ints[{self.mg.int_index(k.value)}]") + if ot is Type.INT: + v = self.ev(e.operand) + t = self._own_new(f"tp_int_box(&{v})") + self.emit(f"tp_int_release(&{v});") + return t if ot not in _BOX: raise CGenError(f"{self.f.name}: Box of {ot}") v = self.ev(e.operand) @@ -1142,16 +1220,26 @@ def ev_Call(self, e: ir.Call) -> str | None: raise CGenError(f"{self.f.name}: Call of {e.function} with {len(e.args)} args") args = [] owned = [] + values = [] # the evaluated arguments, for a redo + int_args = [] # INT arguments of an i64 parameter (redo only) for p, a in zip(callee.params, e.args): - if a.type is not p.type: + int_arg = a.type is Type.INT and p.type is Type.I64 and e.redo + if a.type is not p.type and not int_arg: raise CGenError(f"{self.f.name}: Call {e.function} arg {p.name}: {a.type} for {p.type}") v = self.ev(a) - args.append(v) + values.append(v) + if int_arg: + args.append(f"{v}.v") + int_args.append(v) + else: + args.append(v) if a.type is Type.OBJ: owned.append(v) - values = list(args) # the evaluated arguments, for a redo - t = None + t = ti = None self.emit("tp_s = 0;") + for v in int_args: + # a promoted INT does not fit the callee's i64 parameter: the callee's deopt, redone below + self.emit(f"if ({v}.big != NULL) tp_s = 1;") # The callee's entry globals (ir.Function.entry_globals) are its entry guards: read them # here, at the callee's entry. A mismatch is the callee's deopt (tp_s = 1), handled below # exactly as a deopt inside the callee. @@ -1160,6 +1248,11 @@ def ev_Call(self, e: ir.Call) -> str | None: if callee.returns is not Type.NONE: t = self.tmp(_CT[callee.returns]) args.append(f"&{t}") + if e.type is Type.INT: + if callee.returns is not Type.I64 or not e.redo: + raise CGenError(f"{self.f.name}: Call {e.function} has type INT but is not a redo " + "Call of an i64 function") + ti = self.tmp("tp_int") self.emit("if (tp_s == 0) {") self.depth += 1 # Depth is counted by the callee itself (tp_enter_call at its entry, issue #57). @@ -1170,15 +1263,19 @@ def ev_Call(self, e: ir.Call) -> str | None: self.emit(f"tp_s = {_ident('tp_impl', e.function)}(tp_module{''.join(', ' + x for x in args)});") self.depth -= 1 self.emit("}") + if ti is not None: + self.emit(f"if (tp_s == 0) {{ {ti}.v = {t}; {ti}.big = NULL; }}") if e.redo: - self._redo_call(callee, e.args, values, t) + self._redo_call(callee, e.args, values, t, ti) elif not self.f.pure and callee.pure and self.mg.uses(callee): # a pure callee using classes returns 1 when a class check fails (module docstring, # rule 7): redo it interpreted; this impure caller must not deopt self._redo_call(callee, e.args, values, t) self._own_release(*owned) + for v in int_args: + self.emit(f"tp_int_release(&{v});") self.emit("if (tp_s != 0) { tp_rc = tp_s; goto tp_exit; }") - return t + return ti if ti is not None else t def _read_entry_global(self, g: ir.Param, on_mismatch: str, on_error: str = "tp_rc = -1; goto tp_exit;", guard: str = "tp_s == 0") -> str: @@ -1202,7 +1299,8 @@ def _read_entry_global(self, g: ir.Param, on_mismatch: str, on_error: str = "tp_ self.emit("}") return gv - def _redo_call(self, callee: ir.Function, arg_exprs, values: list[str], t: str | None) -> None: + def _redo_call(self, callee: ir.Function, arg_exprs, values: list[str], t: str | None, + ti: str | None = None) -> None: """ir.Call.redo: on the callee's deopt (tp_s == 1) run `__typedpython_interpreted__[callee]` on the already-evaluated arguments (boxed), count the deopt, convert the result back, and continue in the caller. The callee is pure, so the redo is unobservable.""" @@ -1214,6 +1312,10 @@ def _redo_call(self, callee: ir.Function, arg_exprs, values: list[str], t: str | for a, v in zip(arg_exprs, values): if a.type is Type.OBJ: av.append(v) # still owned by the caller; released after + elif a.type is Type.INT: + b = self._own_new(f"tp_int_box(&{v})") + boxes.append(b) + av.append(b) else: b = self._own_new(f"{_BOX[a.type]}({v})") boxes.append(b) @@ -1239,6 +1341,10 @@ def _redo_call(self, callee: ir.Function, arg_exprs, values: list[str], t: str | elif rt is Type.OBJ: self.emit(f"{t} = {rr}; {rr} = NULL; tp_s = 0;") # moved: the temp owns it now what = "an object" + elif ti is not None: + # SPEC N-13: the redone value is any int: native, or kept as the PyLong it is + self.emit(f"tp_s = tp_int_of_exact(&{ti}, {rr});") + what = "an int" else: self.emit(f"tp_s = {_UNBOX[rt]}({rr}, &{t});") what = {Type.F64: "a float", Type.BOOL: "True or False", Type.I64: "an int in i64"}[rt] @@ -1395,8 +1501,11 @@ def _class(self, name: str) -> int: return self.mg.class_index[name] def ev_Global(self, e: ir.Global) -> str: + # LOAD_GLOBAL through the per-name cache, revalidated by the dict-watcher epoch (SPEC N-13) k = self.mg.name_index(e.name) - return self._own_new(f"tp_global(tp_dict, tp_st->names[{k}])") + g = self.mg.gslot_index(e.name) + return self._own_new(f"tp_gcache_global(&tp_st->gc[{g}], tp_dict, tp_st->builtins, " + f"tp_st->names[{k}], tp_st->watch_id1 > 0)") def ev_GetAttr(self, e: ir.GetAttr) -> str: if e.obj.type is not Type.OBJ: @@ -1605,6 +1714,8 @@ def st_Assign(self, s: ir.Assign) -> None: v = self.ev(s.value) if t is Type.OBJ: self._own_move(_ident("l", s.target), v) + elif t is Type.INT: + self.emit(f"tp_int_assign(&{_ident('l', s.target)}, &{v});") else: self.emit(f"{_ident('l', s.target)} = {v};") @@ -1692,6 +1803,8 @@ def st_ExprStmt(self, s: ir.ExprStmt) -> None: return if s.value.type is Type.OBJ: self._own_release(v) + elif s.value.type is Type.INT: + self.emit(f"tp_int_release(&{v});") else: self.emit(f"(void){v};") @@ -1860,11 +1973,12 @@ def impl(self) -> str: for name, t in self.types.items(): if name in params and t is not Type.OBJ: continue - init = "NULL" if t is Type.OBJ else "0" + init = "NULL" if t is Type.OBJ else "{0, NULL}" if t is Type.INT else "0" ct = _CT[t] out.append(f" {ct}{'' if ct.endswith('*') else ' '}{_ident('l', name)} = {init};") for ct, n in self.temps: - init = "NULL" if ct == "PyObject *" else "{0}" if ct in _CT_ARRAY_REV else "0" + init = ("NULL" if ct == "PyObject *" else "{0}" if ct in _CT_ARRAY_REV + else "{0, NULL}" if ct == "tp_int" else "0") sep = "" if ct.endswith("*") else " " out.append(f" {ct}{sep}{n} = {init};") # (a fast twin declares tp_st / tp_dict only when its body reads them: PyModule_GetState and @@ -2285,6 +2399,8 @@ def _state_functions(mg: _ModGen) -> str: " size_t i;", " if (tp_st == NULL) return 0;", " Py_VISIT(tp_st->breaker);", + " Py_VISIT(tp_st->builtins);", + " for (i = 0; i < sizeof(tp_st->ints) / sizeof(tp_st->ints[0]); i++) Py_VISIT(tp_st->ints[i]);", " Py_VISIT(tp_st->mtype);", " for (i = 0; i < sizeof(tp_st->mdescrs) / sizeof(tp_st->mdescrs[0]); i++) Py_VISIT(tp_st->mdescrs[i]);", " for (i = 0; i < 1; i++) Py_VISIT(tp_st->buf_types[i]);", @@ -2301,8 +2417,12 @@ def _state_functions(mg: _ModGen) -> str: " tp_state *tp_st = (tp_state *)PyModule_GetState(tp_module);", " size_t i;", " if (tp_st == NULL) return 0;", + " tp_globals_unwatch_builtins(tp_st->builtins, tp_st->watch_id1);", " if (tp_st->watch_id1 > 0) tp_globals_unwatch(PyModule_GetDict(tp_module), &tp_st->watch_id1);", " tp_release(&tp_st->breaker);", + " tp_release(&tp_st->builtins);", + " for (i = 0; i < sizeof(tp_st->ints) / sizeof(tp_st->ints[0]); i++) tp_release(&tp_st->ints[i]);", + " for (i = 0; i < sizeof(tp_st->gc) / sizeof(tp_st->gc[0]); i++) tp_gcache_clear(&tp_st->gc[i]);", " for (i = 0; i < sizeof(tp_st->mdescrs) / sizeof(tp_st->mdescrs[0]); i++) tp_release(&tp_st->mdescrs[i]);", " tp_release(&tp_st->mtype);", " for (i = 0; i < 1; i++) tp_release(&tp_st->buf_types[i]);", @@ -2381,6 +2501,9 @@ def _exec_function(mg: _ModGen) -> str: items = ", ".join(f"tp_st->names[{mg.name_index(x)}]" for x in kw) L.append(f" tp_st->kwnames[{k}] = PyTuple_Pack({len(kw)}, {items});") L.append(f" if (tp_st->kwnames[{k}] == NULL) goto tp_done;") + for k, v in enumerate(mg.int_consts): + L.append(f" tp_st->ints[{k}] = PyLong_FromLongLong({_i64_lit(v)});") + L.append(f" if (tp_st->ints[{k}] == NULL) goto tp_done;") L += [ " tp_k = PyDict_ContainsString(tp_dict, \"__builtins__\");", " if (tp_k < 0) goto tp_done;", @@ -2391,6 +2514,17 @@ def _exec_function(mg: _ModGen) -> str: " tp_release(&tp_b);", " if (tp_k < 0) goto tp_done;", " }", + " { /* the builtins dict the module's functions run with (cached builtin reads, SPEC N-13) */", + " PyObject *tp_b = NULL;", + " tp_k = PyDict_GetItemStringRef(tp_dict, \"__builtins__\", &tp_b);", + " if (tp_k < 0) goto tp_done;", + " if (tp_k > 0 && PyModule_Check(tp_b)) {", + " PyObject *tp_bd = Py_NewRef(PyModule_GetDict(tp_b));", + " tp_release(&tp_b);", + " tp_b = tp_bd;", + " }", + " if (tp_k > 0 && tp_b != NULL && PyDict_Check(tp_b)) tp_st->builtins = tp_b; else tp_release(&tp_b);", + " }", " tp_code = Py_CompileString(\"lambda: None\", \"\", Py_eval_input);", " if (tp_code == NULL) goto tp_done;", " tp_st->breaker = PyEval_EvalCode(tp_code, tp_dict, tp_dict);", @@ -2449,10 +2583,17 @@ def _exec_function(mg: _ModGen) -> str: f"tp_st->cls_ok[{j}] = 0;", " } while (0);", ] + if mg.needs_watch(): + # after the source ran: from here a rebinding of a class name or of a cached global moves + # tp_globals_epoch; the builtins dict is watched by the same watcher (cached builtin reads) + L += [ + " tp_globals_watch(tp_dict, tp_watch_cb, &tp_st->watch_id1);", + " if (tp_globals_watch_builtins(tp_st->builtins, tp_st->watch_id1) < 0) {", + " /* cannot watch the builtins: a patched builtin would go unseen, so cache nothing */", + " tp_globals_unwatch(tp_dict, &tp_st->watch_id1);", + " }", + ] if mg.module.classes: - if mg.watches_globals(): - # after the source ran: from here a rebinding of a class name moves tp_globals_epoch - L.append(" tp_globals_watch(tp_dict, tp_watch_cb, &tp_st->watch_id1);") L += [ " tp_cf = PyCFunction_NewEx(&tp_class_info_def, tp_module, NULL);", " if (tp_cf == NULL || PyDict_SetItemString(tp_dict, \"__typedpython_class_info__\", tp_cf) < 0) goto tp_done;", diff --git a/python-multiplatform-ksp/src/main/python/typedpython/cost.py b/python-multiplatform-ksp/src/main/python/typedpython/cost.py index 55fde944..7d2f1f1e 100644 --- a/python-multiplatform-ksp/src/main/python/typedpython/cost.py +++ b/python-multiplatform-ksp/src/main/python/typedpython/cost.py @@ -11,10 +11,12 @@ The rule, applied after verification (so it only ever removes verified functions, never changes the IR): a function is kept when it is **pure**, or when some loop (`While`, `ForRange`; a `ForObj` -is a loop over objects and does not count by itself, but loops nested inside it do) has a body, nested -statements included, containing a native-typed operation: +is a loop over objects and does not count by itself, but loops nested inside it do, and so does an +operation on `INT`s directly in its body, SPEC N-13) has a body, nested statements included, containing +a native-typed operation: - * a `BinOp`, `Compare`, `UnaryOp` or `MathCall` on I64/F64/BOOL operands; + * a `BinOp`, `Compare`, `UnaryOp` or `MathCall` on I64/F64/BOOL/INT operands (an `INT` is a native + i64 promoted in place on overflow, SPEC N-13); * an `Index` or `StoreIndex` on an array or buffer; * a `Call` of a compiled function with at least one argument, all of them native. @@ -33,8 +35,9 @@ interpreter, and that keeps every function that compiled before #150 compiled (fixed-layout setters, Kotlin calls). The rule targets loops over objects only. -The real fix is native ints in impure functions (a separate issue); when it lands, this rule is the -thing to revisit. +Native ints in impure functions (SPEC N-13, #232) landed: an `int` local of an impure function is an +`INT`, so the loop of wordfreq's `word_of` does native work and is kept. What the rule still rejects is +a loop whose body is only object operations. """ from __future__ import annotations @@ -48,7 +51,8 @@ REASON = ("only object operations in its loops: compiled C would be slower than the interpreter " "(#150)") -_NATIVE = (Type.I64, Type.F64, Type.BOOL) +# INT (SPEC N-13, #232) counts: its arithmetic and comparisons are native while the value fits i64 +_NATIVE = (Type.I64, Type.F64, Type.BOOL, Type.INT) def _walk(x): @@ -83,6 +87,25 @@ def _loops(body): yield n +def _int_op(n) -> bool: + """An operation on `INT`s (SPEC N-13): native until a value leaves i64, then CPython's own.""" + if isinstance(n, ir.BinOp): + return n.left.type is Type.INT + if isinstance(n, ir.UnaryOp): + return n.operand.type is Type.INT + if isinstance(n, ir.Compare): + return n.left.type is Type.INT + if isinstance(n, ir.Call): + return any(a.type is Type.INT for a in n.args) and all(a.type in _NATIVE for a in n.args) + return False + + +def _obj_loops(body): + for n in _walk(body): + if isinstance(n, ir.ForObj): + yield n + + def does_native_work(fn: ir.Function) -> bool: if fn.pure: return True @@ -90,7 +113,11 @@ def does_native_work(fn: ir.Function) -> bool: # straight-line code (a field setter, a Kotlin call): no loop to lose to the interpreter, and # what compiled before #150 keeps compiling return True - return any(_native_op(n) for loop in _loops(fn.body) for n in _walk(loop.body)) + if any(_native_op(n) for loop in _loops(fn.body) for n in _walk(loop.body)): + return True + # a loop over an object does not count by itself, nor for float or array work in its body (#150); + # but `INT` arithmetic in it does (SPEC N-13): that is the work the interpreter did slower + return any(_int_op(n) for loop in _obj_loops(fn.body) for n in _walk(loop.body)) def apply(module: ir.Module, diagnostics: list[Diagnostic]) -> tuple[ir.Module, list[Diagnostic]]: diff --git a/python-multiplatform-ksp/src/main/python/typedpython/frontend.py b/python-multiplatform-ksp/src/main/python/typedpython/frontend.py index 5c852322..c9956b33 100644 --- a/python-multiplatform-ksp/src/main/python/typedpython/frontend.py +++ b/python-multiplatform-ksp/src/main/python/typedpython/frontend.py @@ -92,6 +92,7 @@ TYPE_NAME = { Type.I64: "int", Type.F64: "float", Type.BOOL: "bool", Type.NONE: "None", Type.OBJ: "object", + Type.INT: "int", Type.F64_ARRAY: "list[float]", Type.I64_ARRAY: "list[int]", } SCALAR_KIND = {"int": Type.I64, "float": Type.F64, "bool": Type.BOOL} @@ -100,6 +101,7 @@ BINOP = { ast.Add: ir.BinOpKind.ADD, ast.Sub: ir.BinOpKind.SUB, ast.Mult: ir.BinOpKind.MUL, ast.Div: ir.BinOpKind.TRUEDIV, ast.FloorDiv: ir.BinOpKind.FLOORDIV, ast.Mod: ir.BinOpKind.MOD, + ast.LShift: ir.BinOpKind.LSHIFT, ast.RShift: ir.BinOpKind.RSHIFT, } COMPARE = { ast.Eq: ir.CompareKind.EQ, ast.NotEq: ir.CompareKind.NE, ast.Lt: ir.CompareKind.LT, @@ -767,25 +769,39 @@ def __init__(self, var: str) -> None: self.var = var +class _ObjInt(Exception): + """A local of an impure function that would be an `INT` (SPEC N-13) is assigned an object (a call + of an unknown callable, a global, `int(x)`): it stays an object. Raised once per variable; the + function is lowered again with the variable an object.""" + + def __init__(self, var: str) -> None: + super().__init__(var) + self.var = var + + def _lower_with_bounds(sig, env, info, inferred, impure: bool, candidates: dict[str, Type]) -> ir.Function: obj_ranges: frozenset[str] = frozenset() + obj_ints: frozenset[str] = frozenset() while True: try: - return _lower_rounds(sig, env, info, inferred, impure, candidates, obj_ranges) + return _lower_rounds(sig, env, info, inferred, impure, candidates, obj_ranges, obj_ints) except _ObjRange as r: assert r.var not in obj_ranges, "an object range variable was asked for twice" obj_ranges |= {r.var} + except _ObjInt as r: + assert r.var not in obj_ints, "an object int variable was asked for twice" + obj_ints |= {r.var} def _lower_rounds(sig, env, info, inferred, impure: bool, candidates: dict[str, Type], - obj_ranges: frozenset[str]) -> ir.Function: + obj_ranges: frozenset[str], obj_ints: frozenset[str]) -> ir.Function: # Range-target intervals: least fixpoint by iteration, widened to TOP if it does not settle. # A local that keeps growing (a counter) is widened to the i64 limit from round 3 on, as the # verifier's own fixpoint does (verify._prove). bounds: dict[str, Interval] = {} for rnd in range(16): - lowerer = _Lowerer(sig, env, info, inferred, impure, bounds, candidates, obj_ranges) + lowerer = _Lowerer(sig, env, info, inferred, impure, bounds, candidates, obj_ranges, obj_ints) try: fn = lowerer.run() except Skip as e: @@ -803,7 +819,7 @@ def _lower_rounds(sig, env, info, inferred, impure: bool, candidates: dict[str, return fn bounds = joined widened = {var: TOP for var in bounds} - return _Lowerer(sig, env, info, inferred, impure, widened, candidates, obj_ranges).run() + return _Lowerer(sig, env, info, inferred, impure, widened, candidates, obj_ranges, obj_ints).run() def _join(a: Interval, b: Interval) -> Interval: @@ -824,9 +840,11 @@ class _Lowerer: def __init__(self, sig: _Signature, env: dict[str, ir.Function], info: _ModuleInfo, inferred: dict[str, list[tuple[int, str]]], impure: bool, bounds: dict[str, Interval], candidates: dict[str, Type], - obj_ranges: frozenset[str] = frozenset()) -> None: + obj_ranges: frozenset[str] = frozenset(), + obj_ints: frozenset[str] = frozenset()) -> None: self.sig = sig self.obj_ranges = obj_ranges + self.obj_ints = obj_ints self.candidates = candidates self.entry_used: dict[str, Type] = {} self.fn = sig.node @@ -1029,7 +1047,12 @@ def local_type(self, name: str, line: int) -> Type: raise Skip(kinds[0][0], f"local `{name}: {kind}`: a list local is a native array only " "when created by `list(range(n))`, `[v] * n` or `src[:]`") if kind == "int": - t = Type.OBJ if self.impure_mode and name not in self.range_vars else Type.I64 + if not self.impure_mode or name in self.range_vars: + t = Type.I64 + elif name in self.iter_vars or name in self.obj_ints: + t = Type.OBJ # a loop over an object binds objects; an int fed from an object stays one + else: + t = Type.INT # SPEC N-13: native, promoted in place on overflow else: t = SCALAR_KIND.get(kind, Type.OBJ) if name in self.iter_vars and t != Type.OBJ: @@ -1149,6 +1172,8 @@ def assign(self, target: ast.expr, value_node: ast.expr, node: ast.stmt) -> list raise Skip(line, f"array parameter `{name}` is rebound") value = self.value(value_node) t = self.local_type(name, line) + if t == Type.INT and value.type == Type.OBJ: + raise _ObjInt(name) # fed from an object: the local stays one (SPEC N-13) value = self.coerce(value, t, value_node, f"local `{name}`") self.note_assign(name, value) return self.take() + [ir.Assign(name, value)] @@ -1323,7 +1348,7 @@ def new_array(self, name: str, value_node: ast.expr, node: ast.stmt) -> list[ir. length = self.value(value_node.right) if length.type != Type.I64: why = (" (it is a CPython int here: its arithmetic is not proved free of i64 " - "overflow in an impure function)" if length.type == Type.OBJ + "overflow in an impure function)" if length.type in (Type.OBJ, Type.INT) and self.impure_mode else "") raise Skip(line, f"array length `{_src(value_node.right)}` is " f"{TYPE_NAME[length.type]}, not a native int{why}") @@ -1350,6 +1375,8 @@ def aug_assign(self, node: ast.AugAssign) -> list[ir.Stmt]: t = self.local_type(name, line) current = ir.Local(t, name) result = self.arith(op, current, self.sub(node.value), node, inplace=True) + if t == Type.INT and result.type == Type.OBJ: + raise _ObjInt(name) result = self.coerce(result, t, node, f"local `{name}`") self.note_assign(name, result) return self.take() + [ir.Assign(name, result)] @@ -1461,11 +1488,11 @@ def for_range(self, node: ast.For) -> list[ir.Stmt]: def range_bound(self, node: ast.expr, var: str | None = None) -> ir.Expr: e = self.value(node) - if e.type == Type.OBJ and self.impure_mode and var is not None: + if e.type in (Type.OBJ, Type.INT) and self.impure_mode and var is not None: raise _ObjRange(var) # a CPython int object: the loop is a ForObj over range(...) if e.type != Type.I64: why = (" (it is a CPython int here: its arithmetic is not proved free of i64 overflow " - "in an impure function)" if e.type == Type.OBJ and self.impure_mode else "") + "in an impure function)" if e.type in (Type.OBJ, Type.INT) and self.impure_mode else "") raise Skip(node.lineno, f"range bound `{_src(node)}` is not a native int{why}") return e @@ -1535,6 +1562,9 @@ def truth(self, e: ir.Expr, node: ast.expr) -> ir.Expr: return ir.Compare(Type.BOOL, ir.CompareKind.NE, e, ir.Const(Type.I64, 0)) if e.type == Type.F64: return ir.Compare(Type.BOOL, ir.CompareKind.NE, e, ir.Const(Type.F64, 0.0)) + if e.type == Type.INT: + return ir.Compare(Type.BOOL, ir.CompareKind.NE, e, + ir.ToInt(Type.INT, ir.Const(Type.I64, 0))) if e.type == Type.OBJ: self.impure = True return ir.Truth(Type.BOOL, e) @@ -1675,7 +1705,7 @@ def index(self, node: ast.expr) -> ir.Expr: e = self.value(node) if e.type != Type.I64: why = (" (it is a CPython int here: its arithmetic is not proved free of i64 overflow " - "in an impure function)" if e.type == Type.OBJ and self.impure_mode else "") + "in an impure function)" if e.type in (Type.OBJ, Type.INT) and self.impure_mode else "") raise Skip(node.lineno, f"index `{_src(node)}` is {TYPE_NAME[e.type]}, not a native " f"int{why}") return e @@ -1686,6 +1716,10 @@ def binop_kind(self, op: ast.operator, node: ast.AST) -> ir.BinOpKind: raise Skip(node.lineno, f"operator `{_src(node)}` is not supported") return kind + def to_int(self, e: ir.Expr) -> ir.Expr: + """An I64 widened to an INT (SPEC N-13); an INT as it is.""" + return ir.ToInt(Type.INT, e) if e.type == Type.I64 else e + def arith(self, op: ir.BinOpKind, left: ir.Expr, right: ir.Expr, node: ast.AST, inplace: bool = False) -> ir.Expr: lt, rt = left.type, right.type @@ -1694,13 +1728,29 @@ def arith(self, op: ir.BinOpKind, left: ir.Expr, right: ir.Expr, node: ast.AST, f"in `{_src(node)}` is not supported") if Type.OBJ in (lt, rt): return self.obj_binop(op, left, right, inplace) + shift = op in (ir.BinOpKind.LSHIFT, ir.BinOpKind.RSHIFT) + if Type.INT in (lt, rt): + if Type.F64 in (lt, rt) or op == ir.BinOpKind.TRUEDIV: + return self.obj_binop(op, left, right, inplace) # a float result: the object operation + return ir.BinOp(Type.INT, op, self.to_int(left), self.to_int(right)) + if shift: + if Type.F64 in (lt, rt): + return self.obj_binop(op, left, right, inplace) # CPython's TypeError + if not self.impure_mode: + raise Skip(node.lineno, f"`{_src(node)}`: a shift of ints is compiled only in an " + "impure function (native ints with promotion, SPEC N-13)") + return ir.BinOp(Type.INT, op, self.to_int(left), self.to_int(right)) if lt == Type.I64 and rt == Type.I64: result = Type.F64 if op == ir.BinOpKind.TRUEDIV else Type.I64 e = ir.BinOp(result, op, left, right) if self.recordable(e) and self.proved(e): return dataclasses.replace(e, proven=True) if self.impure_mode: - return self.obj_binop(op, left, right, inplace) + if op == ir.BinOpKind.TRUEDIV: + return self.obj_binop(op, left, right, inplace) + # not proved free of overflow, and an impure function cannot deopt: an INT, native + # until an operation leaves i64, then a PyLong, with CPython's results (SPEC N-13) + return ir.BinOp(Type.INT, op, self.to_int(left), self.to_int(right)) self.deopts = True return e if lt == Type.I64: @@ -1724,14 +1774,20 @@ def unary(self, node: ast.UnaryOp) -> ir.Expr: kind = ir.UnaryOpKind.NEG if isinstance(node.op, ast.USub) else ir.UnaryOpKind.POS if operand.type == Type.F64: return ir.UnaryOp(Type.F64, kind, operand) + if operand.type == Type.INT: + return ir.UnaryOp(Type.INT, kind, operand) if operand.type == Type.I64: + if isinstance(operand, ir.Const) and I64_MIN <= -operand.value <= I64_MAX: + # a negative literal is one constant, as CPython's compiler folds it (so a Box of it + # is the module-state constant, SPEC N-13) + return ir.Const(Type.I64, -operand.value if kind == ir.UnaryOpKind.NEG else operand.value) e = ir.UnaryOp(Type.I64, kind, operand) if kind == ir.UnaryOpKind.POS: return e if self.proved(e): return dataclasses.replace(e, proven=True) - if self.impure_mode: # -x of a CPython int is 0 - x - return self.obj_binop(ir.BinOpKind.SUB, ir.Const(Type.I64, 0), operand) + if self.impure_mode: # -x of an i64 may leave i64: an INT (SPEC N-13) + return ir.UnaryOp(Type.INT, kind, self.to_int(operand)) self.deopts = True return e raise Skip(node.lineno, f"unary `{_src(node)}` on {TYPE_NAME[operand.type]} is not " @@ -1784,6 +1840,12 @@ def link(self, op: ast.cmpop, left: ir.Expr, right: ir.Expr, obj_ok: bool, raise Skip(node.lineno, f"comparison `{_src(node)}` is not supported " "(only == != < <= > >=, in, not in, is, is not)") lt, rt = left.type, right.type + if Type.INT in (lt, rt) and Type.OBJ not in (lt, rt) and Type.F64 not in (lt, rt) \ + and Type.BOOL not in (lt, rt): + return ir.Compare(Type.BOOL, kind, self.to_int(left), self.to_int(right)) + if Type.INT in (lt, rt) and Type.OBJ not in (lt, rt) and Type.BOOL not in (lt, rt): + lt = rt = Type.OBJ # an int against a float: compared as objects, like the interpreter + left, right = _box(left), _box(right) if Type.OBJ in (lt, rt): if not obj_ok: raise Skip(node.lineno, f"comparison `{_src(node)}` with an object is not " @@ -1882,11 +1944,13 @@ def direct_call(self, name: str, node: ast.Call) -> ir.Expr | None: self.hoist_ok = False args = [self.value(a) for a in node.args] redo = callee.may_deopt and self.impure_mode and callee.pure \ - and callee.returns in (Type.F64, Type.BOOL, Type.NONE) - # else a deopt could not be redone (an I64 result of a redo could be a big int) + and callee.returns in (Type.F64, Type.BOOL, Type.NONE, Type.I64) + # (an I64 result of a redo may be a big int: the call is then an INT, SPEC N-13) direct = redo or not (callee.may_deopt and self.impure_mode) for p, e in zip(callee.params, args): if p.type != Type.OBJ and e.type != p.type: + if redo and p.type == Type.I64 and e.type == Type.INT: + continue # a promoted INT does not fit: the callee's deopt, redone direct = False # the callee's guard would deopt; the global keeps CPython's result # A class parameter's guard (`type(x) is C`) fails for an argument not proved to be a C: # that is a deopt of the callee at its entry, redoable only by a pure caller. @@ -1904,7 +1968,8 @@ def direct_call(self, name: str, node: ast.Call) -> ir.Expr | None: self.deopts = True typed = tuple(self.boxed(e, a) if p.type == Type.OBJ else e for p, e, a in zip(callee.params, args, node.args)) - return ir.Call(callee.returns, name, typed, redo=redo) + result = Type.INT if redo and callee.returns == Type.I64 else callee.returns + return ir.Call(result, name, typed, redo=redo) def method_call(self, func: ast.Attribute, node: ast.Call) -> ir.Expr | None: """`obj.m(args)` where `obj` is a local of compiled class C and `C.m` a compiled method: the @@ -1963,10 +2028,14 @@ def boxed(self, e: ir.Expr, node: ast.AST) -> ir.Expr: def coerce(self, value: ir.Expr, target: Type, node: ast.AST, what: str) -> ir.Expr: if value.type == target: return value - if target == Type.OBJ and value.type in ir.SCALARS: + if target == Type.OBJ and (value.type in ir.SCALARS or value.type == Type.INT): return ir.Box(Type.OBJ, value) + if target == Type.INT and value.type == Type.I64: + return ir.ToInt(Type.INT, value) extra = "" - if value.type == Type.OBJ: + if value.type == Type.INT: + extra = " (it may be an int beyond i64 here: it cannot be a native i64)" + elif value.type == Type.OBJ: extra = " (CPython would keep the object as it is; there is no conversion to make)" elif value.type == Type.I64 and target == Type.F64: extra = " (CPython keeps an int there, it does not convert it to float)" @@ -2186,14 +2255,14 @@ def _closed(fn: ir.Function, env: dict[str, ir.Function], seen: frozenset[str] = def _box(e: ir.Expr) -> ir.Expr: - return ir.Box(Type.OBJ, e) if e.type in ir.SCALARS else e + return ir.Box(Type.OBJ, e) if e.type in ir.SCALARS or e.type == Type.INT else e def _trivial(e: ir.Expr) -> bool: """No effect, cannot raise, cannot deopt: may be evaluated in any order, or twice.""" if isinstance(e, (ir.Const, ir.Local, ir.Len)): return True - if isinstance(e, ir.ToFloat): + if isinstance(e, (ir.ToFloat, ir.ToInt)): return _trivial(e.operand) if isinstance(e, ir.UnaryOp): return (e.op == ir.UnaryOpKind.NOT or e.type == Type.F64) and _trivial(e.operand) diff --git a/python-multiplatform-ksp/src/main/python/typedpython/ir.py b/python-multiplatform-ksp/src/main/python/typedpython/ir.py index a5cdef90..d4163030 100644 --- a/python-multiplatform-ksp/src/main/python/typedpython/ir.py +++ b/python-multiplatform-ksp/src/main/python/typedpython/ir.py @@ -65,11 +65,14 @@ class Type(Enum): BOOL = "bool" # a Python bool NONE = "none" # the value None (only as a return type) OBJ = "obj" # any Python object, held as a strong reference (PyObject*) + INT = "int" # a Python int that is a native i64 or, once promoted, a PyLong (SPEC N-13) F64_ARRAY = "f64[]" # a list[float] parameter (ArrayParam) or a local array (NewArray/CopyArray) I64_ARRAY = "i64[]" # a list[int] parameter (ArrayParam) or a local array (NewArray/CopyArray) SCALARS = (Type.I64, Type.F64, Type.BOOL) +# INT is not a scalar: it owns a reference when promoted. It is created only by `ToInt`, by arithmetic +# on INTs, and by a redo `Call` of an i64-returning callee, and it becomes an object only through `Box`. ARRAYS = (Type.F64_ARRAY, Type.I64_ARRAY) @@ -103,6 +106,8 @@ class BinOpKind(Enum): TRUEDIV = "/" FLOORDIV = "//" MOD = "%" + LSHIFT = "<<" # INT and OBJ operands only (SPEC N-13) + RSHIFT = ">>" # INT and OBJ operands only @dataclass(frozen=True) @@ -127,6 +132,12 @@ class BinOp(Expr): FLOORDIV/MOD, CPython's float floor division and modulo (sign of divisor, `-0.0` cases), divisor 0.0 → ZeroDivisionError("float floor division by zero") / ("float modulo by zero"). + INT (SPEC N-13; ADD/SUB/MUL/FLOORDIV/MOD/LSHIFT/RSHIFT, both operands INT, result INT; TRUEDIV of ints + is an OBJ operation on boxed values): the native i64 operation when both operands are native + and the result fits i64, else CPython's PyNumber_* on the two ints (exact ints, so no user code + runs) and the result narrowed back to native when it fits. Never deopts, so it is legal in an + impure function; it can raise (ZeroDivisionError, a negative shift count, OverflowError, + MemoryError) exactly where CPython does. `proven` is never set on it. OBJ: CPython's generic operation on objects (PyNumber_*), never deopts. `inplace` (OBJ only) is the augmented form `a op= b` (PyNumber_InPlace*, which a list's `+=` and a user `__iadd__` need); an exact `str + str` is PyUnicode_Concat, the function PyNumber_Add ends in for it. @@ -156,7 +167,8 @@ class UnaryOpKind(Enum): @dataclass(frozen=True) class UnaryOp(Expr): - """NEG of I64 is checked (-(-2**63) → deopt). NOT takes a BOOL and gives a BOOL.""" + """NEG of I64 is checked (-(-2**63) → deopt). NOT takes a BOOL and gives a BOOL. NEG and POS of an + INT give an INT and never deopt (-(-2**63) is the PyLong 2**63).""" op: UnaryOpKind operand: Expr @@ -176,7 +188,8 @@ class CompareKind(Enum): class Compare(Expr): """Result BOOL. Chains (`a < b < c`) are lowered by the front end into `And` with each middle operand evaluated once (via a temporary). Mixed I64/F64 compares **exactly**, as CPython does, - not by converting the int to double (2**53+1 != float(2**53)). NaN compares as in IEEE.""" + not by converting the int to double (2**53+1 != float(2**53)). NaN compares as in IEEE. + Two INT operands compare as the exact ints (native when both are native).""" op: CompareKind left: Expr @@ -206,10 +219,19 @@ class ToFloat(Expr): operand: Expr +@dataclass(frozen=True) +class ToInt(Expr): + """An I64 widened to an INT (type INT): native, no object, never fails (SPEC N-13).""" + + operand: Expr + + @dataclass(frozen=True) class Box(Expr): """A scalar or array element converted to a Python object (type OBJ): I64 → int, F64 → float, - BOOL → True/False. Used when a typed value flows into OBJ code.""" + BOOL → True/False, INT → the int (a promoted INT is the PyLong it holds). Used when a typed value + flows into OBJ code. A Box of an I64 or INT constant is read from module state, built once at + module exec (SPEC N-13).""" operand: Expr @@ -259,7 +281,11 @@ class Call(Expr): `may_deopt`), or the call is `redo`. A pure callee that uses a compiled class can also return the deopt signal with `may_deopt` False, when the class changed after init (N-11); an impure caller then redoes that callee interpreted, for OBJ and I64 results as well, such a callee is pure and - its I64 result is interval-proved, so the redone value is the same and fits.""" + its I64 result is interval-proved, so the redone value is the same and fits. + SPEC N-13: a redo Call of a callee returning I64 has type INT (the redone value may be any int; + it is checked to be exactly an int, anything else is a SystemError), and an argument of type INT + may stand for an I64 parameter: a promoted INT does not fit it, which is the callee's deopt + (absorbed by the redo, the interpreted callee gets the boxed arguments).""" function: str args: tuple[Expr, ...] diff --git a/python-multiplatform-ksp/src/main/python/typedpython/runtime/API.md b/python-multiplatform-ksp/src/main/python/typedpython/runtime/API.md index 6aa9d66f..a6e5ec84 100644 --- a/python-multiplatform-ksp/src/main/python/typedpython/runtime/API.md +++ b/python-multiplatform-ksp/src/main/python/typedpython/runtime/API.md @@ -137,13 +137,53 @@ All return new references or NULL with CPython's exception set; none deopts. PyObject *tp_getattr(PyObject *obj, PyObject *name); /* PyObject_GetAttr */ PyObject *tp_call(PyObject *callee, PyObject *const *args, size_t nargs, PyObject *kwnames); /* PyObject_Vectorcall: nargs = positional count; keyword values follow at args[nargs..] */ /* tp_binop_obj op codes: TP_BINOP_ADD..TP_BINOP_MOD = 0..5 in ir.BinOpKind declaration order; - TP_BINOP_IADD..TP_BINOP_IMOD = 6..11 are the augmented forms (PyNumber_InPlace*, ir.BinOp.inplace) */ + TP_BINOP_IADD..TP_BINOP_IMOD = 6..11 are the augmented forms (PyNumber_InPlace*, ir.BinOp.inplace); + TP_BINOP_LSHIFT, TP_BINOP_RSHIFT = 12, 13 and TP_BINOP_ILSHIFT, TP_BINOP_IRSHIFT = 14, 15 */ int tp_truth(PyObject *obj); /* 0/1, -1 error (PyObject_IsTrue) */ int tp_compare_bool(PyObject *a, PyObject *b, int op); /* 0/1, -1 error: PyObject_RichCompare then PyObject_IsTrue (no identity shortcut, as `if a == b`) */ int tp_obj_to_f64(PyObject *obj, double *out); /* 0 / -1 (PyNumber_Float, then exact double) */ PyObject *tp_binop_obj(PyObject *a, PyObject *b, int op); /* PyNumber_Add/... for ir.BinOpKind on OBJ */ void tp_release(PyObject **slot); /* Py_CLEAR: the only way generated code drops a reference */ +## Native ints (ir.Type.INT; SPEC N-13, issue #232) +A `tp_int` is `{int64_t v; PyObject *big;}`: native when `big == NULL`, else `big` owns a reference to an +exact int outside i64 (`v` unused). Every temporary and local of this type owns `big`; the exit block +releases them all. Each operation is the native one when both operands are native and the result fits +i64, else CPython's `PyNumber_*` on the two ints (exact ints, no user code), with the result narrowed +back to native when it fits. Nothing deopts. Operations return 0, or -1 with the exception set (a zero +divisor, a negative shift count, OverflowError, MemoryError), leaving `*out` untouched on failure. +The overflow checks are the macros `TP_INT_ADD_OVF`, `TP_INT_SUB_OVF`, `TP_INT_MUL_OVF`, the single +mutation points the tests replace by wrapping arithmetic and must see fail. + void tp_int_release(tp_int *a); /* drop big (idempotent) */ + void tp_int_copy(tp_int *dst, const tp_int *src); /* dst = src, a new reference to big */ + void tp_int_assign(tp_int *dst, tp_int *src); /* dst takes src (left empty), releasing what dst held */ + PyObject *tp_int_box(const tp_int *a); /* a new reference to the int (PyLong_FromLongLong when native) */ + int tp_int_take(tp_int *out, PyObject *o); /* o (a new reference, stolen) -> out, narrowed; NULL or a non-exact-int -> -1 */ + int tp_int_of_exact(tp_int *out, PyObject *o); /* o borrowed: 0 ok, 1 not an exact int (nothing set), -1 error */ + int tp_int_add(const tp_int *a, const tp_int *b, tp_int *out); /* also tp_int_sub, tp_int_mul */ + int tp_int_floordiv(const tp_int *a, const tp_int *b, tp_int *out); /* also tp_int_mod: floor semantics */ + int tp_int_lshift(const tp_int *a, const tp_int *b, tp_int *out); /* also tp_int_rshift */ + int tp_int_neg(const tp_int *a, tp_int *out); + int tp_int_cmp(const tp_int *a, const tp_int *b, int op); /* op = Py_EQ..Py_GE; 0 / 1, -1 error */ + +## Global cache (ir.Global; SPEC N-13) +`tp_gslot` is one cached module-global or builtin read: a BORROWED pointer and the epoch it was read at, +so the cache extends no object's lifetime (a rebound function, type or module is freed, and its `__del__`s +run, when CPython would). A read is served from the slot while its `epoch` equals `tp_globals_epoch`; +otherwise `tp_gcache_fill` looks the name up inside the critical sections of the module dict and the +builtins dict captured at exec and remembers the object. The watcher moves the epoch on a change of any +watched name (the names the module reads, the compiled class names and `__builtins__`), a clear or a +dealloc; a rebinding of `__builtins__` also sets `tp_gcache_off`, after which every read does the full +lookup. A hit takes its own reference: on the GIL build `Py_NewRef` (nothing runs between the check and +the incref, and the epoch moves before the dict drops its reference); free-threaded, the fill calls +`PyUnstable_EnableTryIncRef`, a hit uses `PyUnstable_TryIncRef` and then re-reads the epoch, falling back +to the fill when either fails. Free-threaded needs CPython 3.14+ (3.13t does not cache). +`TP_GCACHE_FRESH(slot_epoch, epoch)` is the single mutation point the tests replace by "any nonzero +epoch" and must see fail. Without a watcher (`watched` 0) every read is `tp_global`. + PyObject *tp_gcache_global(tp_gslot *slot, PyObject *module_dict, PyObject *builtins, PyObject *name, int watched); /* a new reference, or NULL with NameError */ + PyObject *tp_gcache_fill(tp_gslot *slot, PyObject *module_dict, PyObject *builtins, PyObject *name); + void tp_gcache_clear(tp_gslot *slot); /* module clear: forget the pointer */ + ## Objects: subscript, in, f-strings, iteration (issue #150; ir.GetItem / StoreItem / Contains / NewList / NewDict / Format / BuildString / ForObj / a str Const) Every helper makes the C API call the interpreter's bytecode makes, so results, exceptions (type, message, args) and the user `__hash__` / `__eq__` / `__index__` / `__getitem__` / `__format__` / @@ -301,6 +341,10 @@ extensions together): /* PyDict_AddWatcher + PyDict_Watch: *id1 = id + 1, or 0 when no id is free / Watch failed (error cleared: the caller uses the conservative path, a dict lookup per New) */ void tp_globals_watch(PyObject *module_dict, PyDict_WatchCallback cb, int *id1); + /* PyDict_Watch(id1 - 1, builtins); 0, or -1 when it cannot be watched (no exception left) */ + int tp_globals_watch_builtins(PyObject *builtins, int id1); + /* PyDict_Unwatch(id1 - 1, builtins); keeps a pending exception */ + void tp_globals_unwatch_builtins(PyObject *builtins, int id1); /* PyDict_Unwatch (if module_dict) + PyDict_ClearWatcher; idempotent; keeps a pending exception */ void tp_globals_unwatch(PyObject *module_dict, int *id1); diff --git a/python-multiplatform-ksp/src/main/python/typedpython/runtime/tp_runtime.h b/python-multiplatform-ksp/src/main/python/typedpython/runtime/tp_runtime.h index 1bd5bc6f..e45eef64 100644 --- a/python-multiplatform-ksp/src/main/python/typedpython/runtime/tp_runtime.h +++ b/python-multiplatform-ksp/src/main/python/typedpython/runtime/tp_runtime.h @@ -136,7 +136,7 @@ static inline void tp_cs_end(tp_cs *c) #define TP_MATH_MSG_FINITE "expected a finite input, got %s" #define TP_MATH_MSG_POSITIVE "expected a positive input, got %s" -/* ir.BinOpKind order, for tp_binop_obj */ +/* ir.BinOpKind (names, not positions), for tp_binop_obj */ #define TP_BINOP_ADD 0 #define TP_BINOP_SUB 1 #define TP_BINOP_MUL 2 @@ -150,6 +150,10 @@ static inline void tp_cs_end(tp_cs *c) #define TP_BINOP_ITRUEDIV 9 #define TP_BINOP_IFLOORDIV 10 #define TP_BINOP_IMOD 11 +#define TP_BINOP_LSHIFT 12 +#define TP_BINOP_RSHIFT 13 +#define TP_BINOP_ILSHIFT 14 +#define TP_BINOP_IRSHIFT 15 /* ------------------------------------------------------------------------------------------ * Integers (int64_t) @@ -1308,6 +1312,10 @@ static inline PyObject *tp_binop_obj(PyObject *a, PyObject *b, int op) case TP_BINOP_ITRUEDIV: return PyNumber_InPlaceTrueDivide(a, b); case TP_BINOP_IFLOORDIV: return PyNumber_InPlaceFloorDivide(a, b); case TP_BINOP_IMOD: return PyNumber_InPlaceRemainder(a, b); + case TP_BINOP_LSHIFT: return PyNumber_Lshift(a, b); + case TP_BINOP_RSHIFT: return PyNumber_Rshift(a, b); + case TP_BINOP_ILSHIFT: return PyNumber_InPlaceLshift(a, b); + case TP_BINOP_IRSHIFT: return PyNumber_InPlaceRshift(a, b); } PyErr_SetString(PyExc_SystemError, "tp_binop_obj: unknown operator"); return NULL; @@ -1488,6 +1496,245 @@ static inline void tp_release(PyObject **slot) Py_CLEAR(*slot); } +/* ------------------------------------------------------------------------------------------ + * Native ints in impure functions (SPEC N-13, ir.Type.INT; API.md "Native ints"). + * A tp_int is a Python int that is a native i64 (`big == NULL`) or, after an operation left the + * i64 range, a PyLong (`big` an owned reference to an exact int, `v` unused). Every operation is + * the native one when both operands are native and the result fits, else CPython's PyNumber_* on + * the two ints (exact ints, so no user code runs) with the result narrowed back to native when it + * fits. Nothing deopts. The generated code owns each tp_int temporary and local, releases `big` on + * every exit and moves a temporary into a local with tp_int_assign. + * ------------------------------------------------------------------------------------------ */ +typedef struct { + int64_t v; + PyObject *big; +} tp_int; + +/* The single mutation points of the overflow checks (the mutation tests in test_native_int_impure.py + * replace each by a wrapping operation and must fail). */ +#define TP_INT_ADD_OVF(a, b, r) __builtin_add_overflow((a), (b), (r)) +#define TP_INT_SUB_OVF(a, b, r) __builtin_sub_overflow((a), (b), (r)) +#define TP_INT_MUL_OVF(a, b, r) __builtin_mul_overflow((a), (b), (r)) + +static inline void tp_int_release(tp_int *a) +{ + if (a->big != NULL) { + Py_CLEAR(a->big); + } +} + +static inline void tp_int_copy(tp_int *dst, const tp_int *src) +{ + dst->v = src->v; + dst->big = src->big != NULL ? Py_NewRef(src->big) : NULL; +} + +/* *dst takes the value of *src (src is left empty); the reference *dst held is released */ +static inline void tp_int_assign(tp_int *dst, tp_int *src) +{ + tp_int old = *dst; + *dst = *src; + src->v = 0; + src->big = NULL; + tp_int_release(&old); +} + +/* a new reference to the int */ +static inline PyObject *tp_int_box(const tp_int *a) +{ + return a->big != NULL ? Py_NewRef(a->big) : PyLong_FromLongLong((long long)a->v); +} + +/* *out = the int `o` (a new reference, stolen): native when it fits i64, else kept as the PyLong. + * NULL (the operation failed) or anything but an exact int -> -1 with the exception set. */ +static inline int tp_int_take(tp_int *out, PyObject *o) +{ + int overflow = 0; + long long v; + if (o == NULL) { + return -1; + } + if (!PyLong_CheckExact(o)) { + Py_DECREF(o); + PyErr_SetString(PyExc_SystemError, "typedpython: an int operation did not produce an int"); + return -1; + } + v = PyLong_AsLongLongAndOverflow(o, &overflow); + if (overflow == 0) { + if (v == -1 && PyErr_Occurred()) { + Py_DECREF(o); + return -1; + } + out->v = (int64_t)v; + out->big = NULL; + Py_DECREF(o); + return 0; + } + out->v = 0; + out->big = o; + return 0; +} + +/* *out = the exact int `o` (borrowed; a new reference is taken when it does not fit i64): + * 0 ok, 1 not an exact int (nothing set, no exception) */ +static inline int tp_int_of_exact(tp_int *out, PyObject *o) +{ + if (!PyLong_CheckExact(o)) { + return 1; + } + return tp_int_take(out, Py_NewRef(o)) == 0 ? 0 : -1; +} + +/* the general case: a op b with CPython's own operator on the two ints */ +static int tp_int_binop_slow(int op, const tp_int *a, const tp_int *b, tp_int *out) +{ + PyObject *x = tp_int_box(a), *y, *r; + if (x == NULL) { + return -1; + } + y = tp_int_box(b); + if (y == NULL) { + Py_DECREF(x); + return -1; + } + r = tp_binop_obj(x, y, op); + Py_DECREF(x); + Py_DECREF(y); + return tp_int_take(out, r); +} + +#define TP_INT_NATIVE2(a, b) ((a)->big == NULL && (b)->big == NULL) + +static inline int tp_int_add(const tp_int *a, const tp_int *b, tp_int *out) +{ + int64_t r; + if (TP_INT_NATIVE2(a, b) && !TP_INT_ADD_OVF(a->v, b->v, &r)) { + out->v = r; + out->big = NULL; + return 0; + } + return tp_int_binop_slow(TP_BINOP_ADD, a, b, out); +} + +static inline int tp_int_sub(const tp_int *a, const tp_int *b, tp_int *out) +{ + int64_t r; + if (TP_INT_NATIVE2(a, b) && !TP_INT_SUB_OVF(a->v, b->v, &r)) { + out->v = r; + out->big = NULL; + return 0; + } + return tp_int_binop_slow(TP_BINOP_SUB, a, b, out); +} + +static inline int tp_int_mul(const tp_int *a, const tp_int *b, tp_int *out) +{ + int64_t r; + if (TP_INT_NATIVE2(a, b) && !TP_INT_MUL_OVF(a->v, b->v, &r)) { + out->v = r; + out->big = NULL; + return 0; + } + return tp_int_binop_slow(TP_BINOP_MUL, a, b, out); +} + +/* floor semantics; a zero divisor and -2**63 // -1 take the general path (CPython's own error / the + * PyLong 2**63) */ +static inline int tp_int_floordiv(const tp_int *a, const tp_int *b, tp_int *out) +{ + int64_t r; + if (TP_INT_NATIVE2(a, b) && b->v != 0 && !(b->v == -1 && a->v == INT64_MIN) + && tp_floordiv_i64(a->v, b->v, &r) == 0) { + out->v = r; + out->big = NULL; + return 0; + } + return tp_int_binop_slow(TP_BINOP_FLOORDIV, a, b, out); +} + +static inline int tp_int_mod(const tp_int *a, const tp_int *b, tp_int *out) +{ + int64_t r; + if (TP_INT_NATIVE2(a, b) && b->v != 0 && tp_mod_i64(a->v, b->v, &r) == 0) { + out->v = r; + out->big = NULL; + return 0; + } + return tp_int_binop_slow(TP_BINOP_MOD, a, b, out); +} + +/* a << b natively for 0 <= b <= 63 when no bit is lost (shifting back gives a); anything else, + * including a negative count (ValueError) and a huge one, is CPython's */ +static inline int tp_int_lshift(const tp_int *a, const tp_int *b, tp_int *out) +{ + if (TP_INT_NATIVE2(a, b) && b->v >= 0 && b->v <= 63) { + int64_t r = (int64_t)((uint64_t)a->v << b->v); + if ((r >> b->v) == a->v) { + out->v = r; + out->big = NULL; + return 0; + } + } + return tp_int_binop_slow(TP_BINOP_LSHIFT, a, b, out); +} + +static inline int tp_int_rshift(const tp_int *a, const tp_int *b, tp_int *out) +{ + if (TP_INT_NATIVE2(a, b) && b->v >= 0) { + out->v = b->v >= 64 ? (a->v < 0 ? -1 : 0) : (a->v >> b->v); + out->big = NULL; + return 0; + } + return tp_int_binop_slow(TP_BINOP_RSHIFT, a, b, out); +} + +static inline int tp_int_neg(const tp_int *a, tp_int *out) +{ + PyObject *x, *r; + if (a->big == NULL && a->v != INT64_MIN) { + out->v = -a->v; + out->big = NULL; + return 0; + } + x = tp_int_box(a); + if (x == NULL) { + return -1; + } + r = PyNumber_Negative(x); + Py_DECREF(x); + return tp_int_take(out, r); +} + +/* a b as a condition: 0 / 1, -1 error. `op` is Py_EQ .. Py_GE. */ +static inline int tp_int_cmp(const tp_int *a, const tp_int *b, int op) +{ + PyObject *x, *y; + int r; + if (TP_INT_NATIVE2(a, b)) { + switch (op) { + case Py_LT: return a->v < b->v; + case Py_LE: return a->v <= b->v; + case Py_EQ: return a->v == b->v; + case Py_NE: return a->v != b->v; + case Py_GT: return a->v > b->v; + default: return a->v >= b->v; + } + } + x = tp_int_box(a); + if (x == NULL) { + return -1; + } + y = tp_int_box(b); + if (y == NULL) { + Py_DECREF(x); + return -1; + } + r = tp_compare_bool(x, y, op); + Py_DECREF(x); + Py_DECREF(y); + return r; +} + /* ------------------------------------------------------------------------------------------ * Fixed-layout classes (SPEC N-11, ir.ClassDecl; API.md "Fixed-layout classes"). A compiled class * is a plain heap type whose instances keep every field in an object slot at a fixed offset; the @@ -1536,6 +1783,111 @@ typedef struct { * one. */ static uint64_t tp_globals_epoch = 1; + +/* ------------------------------------------------------------------------------------------ + * Cached global and builtin reads (SPEC N-13; API.md "Global cache"). LOAD_GLOBAL of a name an + * impure function reads: the object is remembered per name in a tp_gslot and reused while + * tp_globals_epoch has not moved. The epoch moves when the watcher (tp_globals_watch_event) sees + * a change of any watched name in the module dict or the builtins dict, a clear, or a rebinding of + * `__builtins__` (which also turns the cache off for good, tp_gcache_off). + * The slot holds a BORROWED pointer: it extends no lifetime, so a rebound function, type or module + * is freed (and its `__del__`s run) exactly when CPython would free it. A hit must therefore take + * its own reference safely: + * GIL build: nothing runs between the epoch check and the incref, and the epoch moves + * before the dict drops its reference, so an unchanged epoch means the dict + * still holds the object. + * free-threaded: the fill calls PyUnstable_EnableTryIncRef(obj), which delays the object's + * memory past a quiescent state, so PyUnstable_TryIncRef on a just-freed object + * fails safely; after a successful incref the epoch is read again, and an + * unchanged epoch (the watcher bumps it BEFORE the dict store) means the dict + * still held the object when the reference was taken. + * Needs CPython 3.14+ when free-threaded (the TryIncRef API); 3.13t does not cache. + * ------------------------------------------------------------------------------------------ */ +#define TP_GCACHE_FRESH(slot_epoch, epoch) ((slot_epoch) == (epoch)) +#if !defined(Py_GIL_DISABLED) || PY_VERSION_HEX >= 0x030E0000 +#define TP_GCACHE_AVAILABLE 1 +#else +#define TP_GCACHE_AVAILABLE 0 +#endif + +typedef struct { + PyObject *obj; /* BORROWED; ATOMIC, written under the module dict's lock */ + uint64_t epoch; /* tp_globals_epoch at which obj was read; 0 = empty. ATOMIC */ +} tp_gslot; + +static unsigned int tp_gcache_off = 0; /* ATOMIC: 1 after `__builtins__` was rebound */ + +static inline void tp_gcache_clear(tp_gslot *s) +{ + s->epoch = 0; + s->obj = NULL; +} + +/* the slow path: look the name up (module dict, then the builtins dict captured at exec) inside both + * dicts' critical sections, so the epoch read and the lookup cannot straddle a change, and remember + * the (borrowed) object. A new reference, or NULL with the exception (NameError from tp_global when + * the name is unbound). */ +static PyObject *tp_gcache_fill(tp_gslot *slot, PyObject *module_dict, PyObject *builtins, + PyObject *name) +{ + PyObject *v = NULL; + uint64_t epoch = 0; + int rc; + if (!TP_GCACHE_AVAILABLE || builtins == NULL || tp_atomic_load_u32(&tp_gcache_off)) { + return tp_global(module_dict, name); + } + Py_BEGIN_CRITICAL_SECTION2(module_dict, builtins); + epoch = tp_atomic_load_u64(&tp_globals_epoch); + rc = PyDict_GetItemRef(module_dict, name, &v); + if (rc == 0) { + rc = PyDict_GetItemRef(builtins, name, &v); + } + if (rc > 0) { +#if defined(Py_GIL_DISABLED) && PY_VERSION_HEX >= 0x030E0000 + PyUnstable_EnableTryIncRef(v); +#endif + __atomic_store_n(&slot->obj, v, __ATOMIC_RELEASE); /* borrowed: v stays ours to return */ + __atomic_store_n(&slot->epoch, epoch, __ATOMIC_RELEASE); + } + Py_END_CRITICAL_SECTION2(); + if (rc < 0) { + return NULL; + } + if (rc == 0) { + return tp_global(module_dict, name); + } + return v; +} + +/* LOAD_GLOBAL through the cache. `watched`: a dict watcher is installed (without one no change would + * be seen, and every read looks up). */ +static inline PyObject *tp_gcache_global(tp_gslot *slot, PyObject *module_dict, PyObject *builtins, + PyObject *name, int watched) +{ + if (watched && TP_GCACHE_AVAILABLE) { + uint64_t e = __atomic_load_n(&tp_globals_epoch, __ATOMIC_ACQUIRE); + if (TP_GCACHE_FRESH(__atomic_load_n(&slot->epoch, __ATOMIC_ACQUIRE), e)) { + PyObject *p = __atomic_load_n(&slot->obj, __ATOMIC_ACQUIRE); +#if defined(Py_GIL_DISABLED) && PY_VERSION_HEX >= 0x030E0000 + if (p != NULL && PyUnstable_TryIncRef(p)) { + if (__atomic_load_n(&tp_globals_epoch, __ATOMIC_ACQUIRE) == e) { + return p; + } + Py_DECREF(p); + } +#else + if (p != NULL) { + return Py_NewRef(p); + } +#endif + } + } + if (!watched) { + return tp_global(module_dict, name); + } + return tp_gcache_fill(slot, module_dict, builtins, name); +} + /* `name` in cls's OWN dict (not through the MRO). 1 found (*out a new reference), 0 absent, -1 error. */ static inline int tp_own_attr(PyTypeObject *cls, const char *name, PyObject **out) { @@ -1880,6 +2232,9 @@ static inline int tp_globals_watch_event(PyDict_WatchEvent event, PyObject *key, #if PY_VERSION_HEX >= 0x030D0000 for (i = 0; names[i] != NULL; i++) { if (PyUnicode_EqualToUTF8(key, names[i])) { + if (PyUnicode_EqualToUTF8(key, "__builtins__")) { + tp_atomic_store_u32(&tp_gcache_off, 1); /* the builtins dict we watch may be replaced */ + } tp_atomic_add_u64(&tp_globals_epoch, 1); return 0; } @@ -1917,6 +2272,33 @@ static inline void tp_globals_watch(PyObject *module_dict, PyDict_WatchCallback *id1 = id + 1; } +/* Watch the builtins dict with the module's watcher (cached builtin reads, SPEC N-13). Returns 0, or -1 + * when it cannot be watched (no exception left): the caller then drops the watcher, so nothing is cached. */ +static inline int tp_globals_watch_builtins(PyObject *builtins, int id1) +{ + if (id1 <= 0 || builtins == NULL) { + return 0; + } + if (PyDict_Watch(id1 - 1, builtins) < 0) { + PyErr_Clear(); + return -1; + } + return 0; +} + +static inline void tp_globals_unwatch_builtins(PyObject *builtins, int id1) +{ + PyObject *exc; + if (id1 <= 0 || builtins == NULL) { + return; + } + exc = PyErr_GetRaisedException(); + if (PyDict_Unwatch(id1 - 1, builtins) < 0) { + PyErr_Clear(); + } + PyErr_SetRaisedException(exc); +} + /* Undo tp_globals_watch (module clear / free): unwatch module_dict (if still there) and release the * watcher id. Idempotent; keeps any exception that was already set. */ static inline void tp_globals_unwatch(PyObject *module_dict, int *id1) diff --git a/python-multiplatform-ksp/src/main/python/typedpython/verify.py b/python-multiplatform-ksp/src/main/python/typedpython/verify.py index b911e81a..99b25611 100644 --- a/python-multiplatform-ksp/src/main/python/typedpython/verify.py +++ b/python-multiplatform-ksp/src/main/python/typedpython/verify.py @@ -203,8 +203,21 @@ keeps its checked, deopting ops (and is then legal only if pure). Call.redo. Legal only if the caller is impure, the callee is pure and may_deopt, and the callee -returns F64, BOOL or NONE (`verify/redo` otherwise). A redo call is not a deopting node in the -caller: the callee alone is redone, unobservably, since it is pure. +returns F64, BOOL, NONE or I64 (`verify/redo` otherwise); a redo Call of an I64-returning callee has +type INT (below). A redo call is not a deopting node in the caller: the callee alone is redone, +unobservably, since it is pure. + +Native ints (SPEC N-13, #232). `Type.INT` is a Python int that is a native i64 or, once an operation +left the i64 range, a PyLong. It lives only in locals and temporaries (never a parameter, entry global +or return value). It is produced only by `ToInt` (of an I64), by BinOp/UnaryOp over INT operands, by +an INT local, and by a redo Call of an I64-returning callee (an INT argument may stand for an I64 +parameter of a redo callee: a promoted value does not fit, which is the callee's deopt, redone). +BinOp on INT: both operands INT, ADD/SUB/MUL/FLOORDIV/MOD/LSHIFT/RSHIFT, result INT; TRUEDIV of ints is an +OBJ operation on boxed values; `proven` is never legal on it. UnaryOp NEG/POS of INT is INT. Compare of two +INTs is BOOL. An INT crosses into an object slot only through `Box`, and an I64 into an INT slot only +through `ToInt`; an INT never flows into an I64 slot. None of these nodes can deopt (they promote), run +user code or make a function impure, so an impure function may contain them freely, and they do not +affect `may_deopt`, `pure` or closedness. The shifts exist only on INT and OBJ. Function.entry_globals. Scalar `Param`s (I64/F64/BOOL), read once at entry as part of the guards; assigned at entry for definite assignment; the body reads them as `Local(name)` of that type and may @@ -226,7 +239,7 @@ And, ArrayParam, Assign, BinOp, BinOpKind, Box, Break, Call, CallObject, Compare, CompareKind, CompareObj, Const, Continue, CopyArray, ExprStmt, ForRange, Function, GetAttr, Global, If, Index, Len, Local, MathCall, MathFunc, Module, NewArray, ObjToFloat, Or, Param, Return, StoreIndex, ToFloat, - Truth, Tuple, Type, Unbox, UnaryOp, UnaryOpKind, While, + ToInt, Truth, Tuple, Type, Unbox, UnaryOp, UnaryOpKind, While, ) from .ir import CheckExact, CheckMethod, ClassDecl, FieldGet, FieldSet, Is, IsExact, New from .ir import BuildString, Contains, ForObj, Format, GetItem, NewDict, NewList, StoreItem @@ -265,7 +278,7 @@ class Diagnostic: # Exact classes, not isinstance: a subclass of an ir node is not an ir node (its meaning is not # the one the back end implements). -_EXPRS = frozenset({Const, Local, BinOp, UnaryOp, Compare, And, Or, ToFloat, Box, Unbox, MathCall, +_EXPRS = frozenset({Const, Local, BinOp, UnaryOp, Compare, And, Or, ToFloat, ToInt, Box, Unbox, MathCall, Call, Global, GetAttr, CallObject, Truth, CompareObj, ObjToFloat, Len, Index, NewArray, CopyArray, Tuple, Is, IsExact, CheckExact, CheckMethod, FieldGet, New, GetItem, Contains, NewList, NewDict, Format, BuildString}) @@ -280,7 +293,12 @@ class Diagnostic: GetItem, NewList, NewDict, Format, BuildString}) _CONVERSIONS = (-1, 97, 114, 115) # Format.conversion: none, !a, !r, !s _VALUE_TYPES = (Type.I64, Type.F64, Type.BOOL, Type.OBJ) # what a local may hold +_LOCAL_TYPES = _VALUE_TYPES + (Type.INT,) # an INT lives only in a local or a temporary (SPEC N-13) _RETURN_TYPES = _VALUE_TYPES + (Type.NONE,) +# The nodes that produce an INT (an owned tp_int: native, or a PyLong once promoted). +_INT_PRODUCERS = frozenset({ToInt, BinOp, UnaryOp, Local, Call}) +_INT_OPS = frozenset({BinOpKind.ADD, BinOpKind.SUB, BinOpKind.MUL, BinOpKind.FLOORDIV, + BinOpKind.MOD, BinOpKind.LSHIFT, BinOpKind.RSHIFT}) _ELEMENT = {Type.F64_ARRAY: Type.F64, Type.I64_ARRAY: Type.I64} _MATH_ARITY = {MathFunc.ATAN2: 2, MathFunc.HYPOT: 2} # every other MathFunc: 1 _I64_DEOPT_OPS = frozenset({BinOpKind.ADD, BinOpKind.SUB, BinOpKind.MUL, BinOpKind.FLOORDIV, @@ -437,7 +455,7 @@ def declarations(self) -> frozenset[str]: self.error(TYPE, f"local array '{name}' collides with a scalar parameter") else: self.larrays[name] = t - elif t not in _VALUE_TYPES: + elif t not in _LOCAL_TYPES: self.error(TYPE, f"local '{name}' has type {t}") elif name in self.scalars and self.scalars[name] is not t: self.error(TYPE, f"local '{name}' declared {t} but parameter is " @@ -554,7 +572,15 @@ def slot(self, e: ir.Expr, want: Type, what: str) -> None: got = getattr(e, "type", None) if got is want: return - if got is Type.OBJ and want in ir.SCALARS: + if got is Type.INT and want is Type.OBJ: + self.error(OWNERSHIP, f"{what}: int flows into an OBJ slot without Box") + elif want is Type.INT and got in (Type.I64, Type.OBJ): + self.error(TYPE if got is Type.I64 else OWNERSHIP, + f"{what}: {got.value} flows into an int slot without ToInt") + elif got is Type.INT and want in ir.SCALARS: + self.error(TYPE, f"{what}: an int (native or promoted) cannot flow into a {want.value} " + f"slot") + elif got is Type.OBJ and want in ir.SCALARS: self.error(OWNERSHIP, f"{what}: OBJ flows into a {want.value} slot without Unbox") elif want is Type.OBJ and got in ir.SCALARS: self.error(OWNERSHIP, f"{what}: {got.value} flows into an OBJ slot without Box") @@ -580,6 +606,9 @@ def expr(self, e: object, state: _State, cond: bool = False) -> ir.Expr: self.error(OWNERSHIP, f"{cls.__name__} cannot produce an OBJ (only Box, Global, GetAttr, CallObject, OBJ " f"BinOp, Tuple, OBJ locals and OBJ-returning Calls create " f"references)") + if t is Type.INT and cls not in _INT_PRODUCERS: + self.error(OWNERSHIP, f"{cls.__name__} cannot produce an int (only ToInt, int BinOp/UnaryOp, " + f"int locals and redo Calls of i64 functions do)") if cls is CompareObj and not cond: self.error(CONDITION_ONLY, "CompareObj outside a condition: the rich comparison's " "result object would be kept, and it need not be a bool") @@ -742,11 +771,23 @@ def e_BinOp(self, e: BinOp, state: _State) -> ir.Expr: exact = (_exact(e.op, self.interval(left), self.interval(right)) if shape else _FULL) proven = self.check_proven(shape, exact, f"{e.type.value} {e.op.value}") - if lt not in (Type.I64, Type.F64, Type.OBJ): + shift = e.op in (BinOpKind.LSHIFT, BinOpKind.RSHIFT) + if lt is Type.INT: + # SPEC N-13: both operands INT, result INT, never deopts (it promotes in place) + if rt is not Type.INT: + self.slot(right, Type.INT, f"right operand of {e.op.value}") + elif e.op not in _INT_OPS: + self.error(TYPE, f"int {e.op.value} has no INT form (box the operands: it is an OBJ " + f"operation)") + elif e.type is not Type.INT: + self.error(TYPE, f"int {e.op.value} int is int, node says {e.type.value}") + elif lt not in (Type.I64, Type.F64, Type.OBJ): self.error(TYPE if lt is not Type.OBJ else OWNERSHIP, f"BinOp {e.op.value} on {getattr(lt, 'value', lt)} operands") elif lt is not rt: self.slot(right, lt, f"right operand of {e.op.value}") + elif shift and lt is not Type.OBJ: + self.error(TYPE, f"{e.op.value} on {lt.value} operands: only int (INT) and OBJ shift") else: want = Type.F64 if (lt is Type.I64 and e.op is BinOpKind.TRUEDIV) else lt if e.type is not want: @@ -757,6 +798,8 @@ def e_BinOp(self, e: BinOp, state: _State) -> ir.Expr: if lt is Type.OBJ: self.effects.append(f"OBJ {e.op.value} (runs user methods)") self.opens.append(f"OBJ {e.op.value} (runs user methods)") + if e.proven and lt is Type.INT: + self.error(PROVEN, f"`proven` is set on an int {e.op.value}, which promotes instead") return dataclasses.replace(e, left=left, right=right) def e_UnaryOp(self, e: UnaryOp, state: _State) -> ir.Expr: @@ -777,6 +820,12 @@ def e_UnaryOp(self, e: UnaryOp, state: _State) -> ir.Expr: self.slot(operand, Type.BOOL, "operand of not") if e.type is not Type.BOOL: self.error(TYPE, f"not gives bool, node says {e.type.value}") + elif ot is Type.INT: + if e.type is not Type.INT: + self.error(TYPE, f"unary {e.op.value} of int is int, node says {e.type.value}") + if e.proven: + self.error(PROVEN, f"`proven` is set on unary {e.op.value} of an int, which " + f"promotes instead") else: if ot is Type.OBJ: self.error(OWNERSHIP, f"unary {e.op.value} on OBJ has no IR form") @@ -796,7 +845,12 @@ def e_Compare(self, e: Compare, state: _State) -> ir.Expr: left, right = self.expr(e.left, state), self.expr(e.right, state) lt, rt = getattr(left, "type", None), getattr(right, "type", None) numeric = (Type.I64, Type.F64) - if Type.OBJ in (lt, rt): + if lt is Type.INT or rt is Type.INT: + if lt is not rt: + self.error(TYPE, f"Compare of {getattr(lt, 'value', lt)} and " + f"{getattr(rt, 'value', rt)}: both sides of an int comparison are " + f"INT (ToInt widens an i64)") + elif Type.OBJ in (lt, rt): self.error(OWNERSHIP, "Compare on an OBJ operand (rich comparison returns an object, " "not a bool)") elif not ((lt in numeric and rt in numeric) or (lt is Type.BOOL and rt is Type.BOOL)): @@ -830,10 +884,19 @@ def e_ToFloat(self, e: ToFloat, state: _State) -> ir.Expr: self.error(TYPE, f"ToFloat gives f64, node says {e.type.value}") return dataclasses.replace(e, operand=operand) + def e_ToInt(self, e: ToInt, state: _State) -> ir.Expr: + operand = self.expr(e.operand, state) + self.slot(operand, Type.I64, "operand of ToInt") + if e.type is not Type.INT: + self.error(TYPE, f"ToInt gives int, node says {e.type.value}") + return dataclasses.replace(e, operand=operand) + def e_Box(self, e: Box, state: _State) -> ir.Expr: operand = self.expr(e.operand, state) ot = getattr(operand, "type", None) - if ot is Type.OBJ: + if ot is Type.INT: + pass + elif ot is Type.OBJ: self.error(OWNERSHIP, "Box of an OBJ (it is already an object)") elif ot not in ir.SCALARS and ot not in ir.ARRAYS: self.error(TYPE, f"Box of {getattr(ot, 'value', ot)}") @@ -885,14 +948,19 @@ def e_Call(self, e: Call, state: _State) -> ir.Expr: if len(args) != len(callee.params): self.error(TYPE, f"'{e.function}' takes {len(callee.params)} argument(s), got " f"{len(args)}") + redo_ok = type(e.redo) is bool and e.redo and self.fn.pure is not True for k, (a, p) in enumerate(zip(args, callee.params)): if type(p) is ArrayParam: self.error(ARRAY_USE, f"'{e.function}' takes array parameter '{p.name}'; an array " f"cannot be passed by Call (the entry guard and write-back " f"belong to the Python-level call)") + elif (redo_ok and p.type is Type.I64 and getattr(a, "type", None) is Type.INT): + pass # SPEC N-13: a promoted INT does not fit the parameter: the callee's deopt, redone else: self.slot(a, p.type, f"argument {k + 1} of '{e.function}'") - if e.type is not callee.returns: + # SPEC N-13: a redo Call of an i64-returning callee gives an INT (the redone value is any int) + int_result = (redo_ok and callee.returns is Type.I64 and e.type is Type.INT) + if e.type is not callee.returns and not int_result: self.error(TYPE, f"'{e.function}' returns {callee.returns.value}, node says " f"{e.type.value}") redo = False @@ -905,10 +973,13 @@ def e_Call(self, e: Call, state: _State) -> ir.Expr: elif callee.pure is not True or callee.may_deopt is not True: self.error(REDO, f"redo call of '{e.function}', which is not a pure may_deopt " f"function") - elif callee.returns not in (Type.F64, Type.BOOL, Type.NONE): + elif callee.returns not in (Type.F64, Type.BOOL, Type.NONE, Type.I64): self.error(REDO, f"redo call of '{e.function}' returning " f"{getattr(callee.returns, 'value', callee.returns)}: only f64, " - f"bool and none results are exactly reproducible") + f"bool, none and i64 (as an int) results are exactly reproducible") + elif callee.returns is Type.I64 and e.type is not Type.INT: + self.error(REDO, f"redo call of '{e.function}' returning i64 must have type int " + f"(the redone value may be a big int)") else: redo = True if not redo: diff --git a/python-multiplatform-ksp/src/test/python/test_class_methods.py b/python-multiplatform-ksp/src/test/python/test_class_methods.py index 90bb4395..d6693dca 100644 --- a/python-multiplatform-ksp/src/test/python/test_class_methods.py +++ b/python-multiplatform-ksp/src/test/python/test_class_methods.py @@ -913,7 +913,7 @@ def test_binary_trees_with_check_as_a_method_compiles_and_matches(tmp_path): "binary_trees": {"check", "make"}, "fannkuch": {"fannkuch"}, "spectral_norm": {"eval_a", "mul_atv", "mul_av"}, - "wordfreq": {"next_state"}, + "wordfreq": {"next_state", "word_of", "main"}, # word_of and main: native ints, #232 "nbody": set(), } diff --git a/python-multiplatform-ksp/src/test/python/test_cost_rule.py b/python-multiplatform-ksp/src/test/python/test_cost_rule.py index f7411245..db6c4999 100644 --- a/python-multiplatform-ksp/src/test/python/test_cost_rule.py +++ b/python-multiplatform-ksp/src/test/python/test_cost_rule.py @@ -1,8 +1,9 @@ """The cost policy (cost.py, #150): a function is compiled only if it does native work. -A verified function that only does object operations in its loops (in an impure function an int is a -CPython object, so `+`, `//`, `%` and `>` are generic PyNumber / RichCompare calls) is slower compiled -than interpreted, so it stays interpreted with a diagnostic. Pure functions are never touched. +A verified function that only does object operations in its loops (a `str` concatenation, a call of an +unknown callable: generic C API calls) is slower compiled than interpreted, so it stays interpreted +with a diagnostic. Arithmetic on an `int` local of an impure function is native work since #232 (the +IR type INT, SPEC N-13), so such a loop compiles. Pure functions are never touched. `pipeline.compile_module(..., cost_rule=False)` is the test-only opt-out. """ from __future__ import annotations @@ -16,6 +17,15 @@ HEADER = "# typedpython: compiled\n" OBJECT_ONLY = HEADER + textwrap.dedent(''' + def copy_words(src: list[str], dst: list[str]) -> str: + w: str = "" + for x in src: + w = x + w + dst.append(w) + return w +''') + +INT_LOOP = HEADER + textwrap.dedent(''' def word_of(word_id: int) -> str: x: int = word_id + 17576 w: str = "" @@ -39,14 +49,22 @@ def names(result): def test_an_object_only_loop_stays_interpreted_with_the_reason(tmp_path): r = build(tmp_path, OBJECT_ONLY) assert names(r) == set() and r.extension is None - assert "slower than the interpreter" in r.skipped["word_of"] - assert r.skipped["word_of"].startswith("policy/cost: only object operations in its loops") + assert "slower than the interpreter" in r.skipped["copy_words"] + assert r.skipped["copy_words"].startswith("policy/cost: only object operations in its loops") assert [d.rule for d in r.diagnostics] == ["policy/cost"] +def test_an_int_loop_is_native_work_and_compiles(tmp_path): + # the function that #150 had to leave interpreted (wordfreq's word_of): its `x > 0`, `x % 26` + # and `x //= 26` are INT operations now (SPEC N-13) + r = build(tmp_path, INT_LOOP) + assert names(r) == {"word_of"} and r.extension is not None, r.skipped + assert not [d for d in r.diagnostics if d.rule == "policy/cost"] + + def test_the_opt_out_compiles_the_same_function(tmp_path): r = build(tmp_path, OBJECT_ONLY, cost_rule=False) - assert names(r) == {"word_of"} and r.extension is not None + assert names(r) == {"copy_words"} and r.extension is not None MIXED = HEADER + textwrap.dedent(''' @@ -66,7 +84,14 @@ def pure_no_loop(a: float, b: float) -> float: return a * b + 1.0 - def object_loop_with_native_op_outside(n: int, xs: list[str]) -> float: + def object_loop_with_native_op_outside(n: int, xs: list[str], ys: list[str]) -> float: + total: float = 1.0 + 2.0 + for x in xs: + ys.append(x) + return total + + + def int_counter_loop(n: int, xs: list[str]) -> float: total: float = 1.0 + 2.0 i: int = 0 while i < n: @@ -87,7 +112,7 @@ def native_loop_inside_object_loop(rows: list[list[str]]) -> float: def test_one_native_loop_with_object_operations_compiles(tmp_path): r = build(tmp_path, MIXED) got = names(r) - assert {"mixed", "pure_no_loop", "native_loop_inside_object_loop"} <= got, r.skipped + assert {"mixed", "pure_no_loop", "native_loop_inside_object_loop", "int_counter_loop"} <= got, r.skipped assert "object_loop_with_native_op_outside" not in got # the native op is outside the loop assert "object_loop_with_native_op_outside" in r.skipped @@ -103,17 +128,17 @@ def test_a_caller_of_a_rejected_function_is_rejected_too(tmp_path): src = tmp_path / "p.py" src.write_text(OBJECT_ONLY) proved, diags = verify.verify(frontend.lower(src)) - assert [f.name for f in proved.functions] == ["word_of"] + assert [f.name for f in proved.functions] == ["copy_words"] word = proved.functions[0] caller = ir.Function(name="c", params=(), returns=ir.Type.NONE, locals={}, pure=False, may_deopt=False, source_line=1, body=(ir.ForRange("i", ir.Const(ir.Type.I64, 0), ir.Const(ir.Type.I64, 3), ir.Const(ir.Type.I64, 1), - (ir.ExprStmt(ir.Call(ir.Type.NONE, "word_of", ())),)),)) + (ir.ExprStmt(ir.Call(ir.Type.NONE, "copy_words", ())),)),)) module = ir.Module("p", (word, caller)) out, diags = cost.apply(module, []) assert out.functions == () # `c` has no native op (an empty-arg Call is not one) - assert "word_of" in out.skipped and "c" in out.skipped + assert "copy_words" in out.skipped and "c" in out.skipped def test_the_rule_is_what_keeps_it_interpreted(tmp_path, monkeypatch): diff --git a/python-multiplatform-ksp/src/test/python/test_frontend.py b/python-multiplatform-ksp/src/test/python/test_frontend.py index 3a51a355..8b2a55fd 100644 --- a/python-multiplatform-ksp/src/test/python/test_frontend.py +++ b/python-multiplatform-ksp/src/test/python/test_frontend.py @@ -16,7 +16,7 @@ from typedpython.ir import ( And, ArrayParam, Assign, BinOp, BinOpKind, Box, Break, Call, CallObject, Compare, CompareKind, CompareObj, Const, CopyArray, NewArray, Tuple, ExprStmt, While, ForRange, GetAttr, Global, If, Index, Len, Local, MathCall, - MathFunc, ObjToFloat, Param, Return, StoreIndex, ToFloat, Truth, Type, UnaryOp, UnaryOpKind, + MathFunc, ObjToFloat, Param, Return, StoreIndex, ToFloat, ToInt, Truth, Type, UnaryOp, UnaryOpKind, ) I64, F64, BOOL, OBJ = Type.I64, Type.F64, Type.BOOL, Type.OBJ @@ -103,8 +103,9 @@ def f(a: int) -> float: assert "`c`" in m.skipped["f"] and "line " in m.skipped["f"] -def test_impure_function_keeps_an_unbounded_int_local_as_a_python_int(tmp_path): - # Impure: a deopt could not be redone, so `m` is a CPython int object, not a checked i64. +def test_impure_function_keeps_an_unbounded_int_local_as_a_native_int(tmp_path): + # Impure: a deopt could not be redone, so `m` is an INT (SPEC N-13): native i64, promoted in place + # to a PyLong on overflow, not a checked i64. m = lower(tmp_path, """ @compiled def g(a: list[float], n: int) -> None: @@ -113,12 +114,11 @@ def g(a: list[float], n: int) -> None: a[0] = 1.0 """) f = function(m, "g") - assert f.locals == {"m": OBJ} + INT = Type.INT + assert f.locals == {"m": INT} assert f.body[:2] == ( - Assign("m", Box(OBJ, Local(I64, "n"))), - # `m += 1` on an object is the in-place operation (PyNumber_InPlaceAdd, #150) - Assign("m", BinOp(OBJ, BinOpKind.ADD, Local(OBJ, "m"), Box(OBJ, Const(I64, 1)), - inplace=True)), + Assign("m", ToInt(INT, Local(I64, "n"))), + Assign("m", BinOp(INT, BinOpKind.ADD, Local(INT, "m"), ToInt(INT, Const(I64, 1)))), ) assert not f.pure and not f.may_deopt @@ -432,8 +432,8 @@ def fill(n: int, out: list[float]) -> None: assert call.function == "tri" and call.redo -def test_impure_caller_of_an_int_returning_deopting_callee_calls_the_global(tmp_path): - # An I64 result of a redo could be a big int: such a callee is still called as an object. +def test_impure_caller_of_an_int_returning_deopting_callee_redoes_it_as_an_int(tmp_path): + # SPEC N-13: an I64 result of a redo may be a big int, so the direct Call has type INT. m = lower(tmp_path, """ @compiled def add(i: int, j: int) -> int: @@ -446,9 +446,9 @@ def fill(n: int, out: list[float]) -> None: """) fill = function(m, "fill") assert not fill.pure and not fill.may_deopt - assert nodes(fill, Call) == [] - assert nodes(fill, CallObject) == [ - CallObject(OBJ, Global(OBJ, "add"), (Box(OBJ, Local(I64, "i")), Box(OBJ, Local(I64, "i"))))] + assert nodes(fill, CallObject) == [] + assert nodes(fill, Call) == [ + Call(Type.INT, "add", (Local(I64, "i"), Local(I64, "i")), redo=True)] def test_pure_caller_of_a_deopting_callee_does_not_redo(tmp_path): @@ -699,7 +699,7 @@ def render(n: int, scale: float) -> float: """) f = function(m, "render") assert not f.pure and not f.may_deopt - assert f.locals == {"canvas": OBJ, "total": F64, "hits": OBJ, "i": I64, "x": F64} + assert f.locals == {"canvas": OBJ, "total": F64, "hits": Type.INT, "i": I64, "x": F64} assert f.body[0] == Assign("canvas", CallObject( OBJ, Global(OBJ, "make_canvas"), (Box(OBJ, Local(I64, "n")),))) loop = f.body[3] @@ -711,9 +711,9 @@ def render(n: int, scale: float) -> float: ExprStmt(CallObject(OBJ, Global(OBJ, "draw"), ( Local(OBJ, "canvas"), Box(OBJ, Local(F64, "x")), Box(OBJ, Const(F64, 0.5)), ), ("alpha",))), - Assign("hits", BinOp(OBJ, BinOpKind.ADD, Local(OBJ, "hits"), Box(OBJ, Const(I64, 1)), - inplace=True)), # `hits += 1` on an object: in place (#150) - If(CompareObj(BOOL, CompareKind.GT, Local(OBJ, "hits"), Box(OBJ, Local(I64, "n"))), + Assign("hits", BinOp(Type.INT, BinOpKind.ADD, Local(Type.INT, "hits"), + ToInt(Type.INT, Const(I64, 1)))), # native int (SPEC N-13) + If(Compare(BOOL, CompareKind.GT, Local(Type.INT, "hits"), ToInt(Type.INT, Local(I64, "n"))), (Break(),)), ) diff --git a/python-multiplatform-ksp/src/test/python/test_native_int_impure.py b/python-multiplatform-ksp/src/test/python/test_native_int_impure.py new file mode 100644 index 00000000..f9724475 --- /dev/null +++ b/python-multiplatform-ksp/src/test/python/test_native_int_impure.py @@ -0,0 +1,727 @@ +"""Native ints in impure functions (SPEC N-13, #232). + +An impure function cannot deopt after an effect. Its `int` locals are the IR type `INT`: a native +i64 that is promoted in place to a PyLong when an operation leaves the i64 range, with the results of +CPython. Everything here is a differential test against the interpreted function of the same module +(`__typedpython_interpreted__`), so a result, a raised exception, the side effects that happened before +it and the absence of a deopt must all be equal. + +Red before the implementation: the IR has no `INT` type (the structural tests fail with +AttributeError), the C has no per-name global cache and no module-state integer constants, and +`word_of` / `main` of the wordfreq benchmark are rejected by the cost rule. +""" +from __future__ import annotations + +import builtins +import contextlib +import gc +import importlib.util +import io +import re +import shutil +import sys +import textwrap +import threading +import types +import weakref +from pathlib import Path + +import pytest + +from typedpython import cbuild, cgen, cost, frontend, ir, verify + +REPO = Path(__file__).resolve().parents[4] +WORDFREQ = REPO / "benchmarks" / "typedpython" / "py" / "wordfreq.py" + +SOURCE = textwrap.dedent(''' + # typedpython: compiled + + import math as GMOD + + LIMIT: int = 10 + MOD: int = 2147483647 + + + def _one() -> int: + return 1 + + + def _two() -> int: + return 2 + + + class GTYPE: + pass + + + GFN = _one + + + def add2(log: list[object], a: int, b: int) -> object: + log.append("start") + r: int = a + b + log.append(r) + r = r + b + log.append(r) + return r + + + def sub2(log: list[object], a: int, b: int) -> object: + log.append("start") + r: int = a - b + log.append(r) + r = r - b + log.append(r) + return r + + + def mul2(log: list[object], a: int, b: int) -> object: + log.append("start") + r: int = a * b + log.append(r) + r = r * b + log.append(r) + return r + + + def neg2(log: list[object], a: int) -> object: + log.append("start") + r: int = -a + log.append(r) + r = -r + log.append(r) + return r + + + def floordiv2(log: list[object], a: int, b: int) -> object: + log.append("start") + r: int = a + 1 + log.append(r) + r = r // b + log.append(r) + r = a // b + log.append(r) + return r + + + def mod2(log: list[object], a: int, b: int) -> object: + log.append("start") + r: int = a - 1 + log.append(r) + r = r % b + log.append(r) + r = a % b + log.append(r) + return r + + + def lsh(log: list[object], a: int, b: int) -> object: + log.append("start") + r: int = a << b + log.append(r) + return r + + + def rsh(log: list[object], a: int, b: int) -> object: + log.append("start") + r: int = a + 1 + log.append(r) + r = r >> b + log.append(r) + return r + + + def roundtrip(log: list[object], a: int) -> object: + log.append(a) + x: int = a + x = x + 10 + log.append(x) + x = x - 20 + log.append(x) + x = x * 2 + log.append(x) + x = x // 2 + log.append(x) + return x + + + def compare(log: list[object], a: int, b: int) -> None: + log.append("start") + x: int = a + 1 + y: int = b - 1 + if x < y: + log.append("lt") + if x <= y: + log.append("le") + if x > y: + log.append("gt") + if x >= y: + log.append("ge") + if x == y: + log.append("eq") + if x != y: + log.append("ne") + if x: + log.append("truthy") + if not y: + log.append("y-zero") + + + def fact(log: list[object], n: int) -> object: + log.append(n) + f: int = 1 + for i in range(n): + f = f * (i + 1) + log.append(f) + return f + + + def grow(log: list[object], a: int, k: int) -> object: + x: int = a + i: int = 0 + while i < k: + x = x * 3 + 1 + i += 1 + log.append(x) + return x + + + def consts(log: list[object]) -> None: + log.append(7000000000) + log.append(5) + log.append(-7000000000) + + + def next_state(s: int) -> int: + return (s * 48271) % MOD + + + def drive(log: list[object], seed: int, n: int) -> object: + s: int = seed + 1 + for _ in range(n): + s = next_state(s) + log.append(s) + return s + + + def read_limit(log: list[object]) -> object: + log.append(1) + return LIMIT + + + def read_twice(log: list[object], cb: object) -> None: + log.append(LIMIT) + cb() + log.append(LIMIT) + + + def use_builtin(log: list[object], s: str) -> object: + log.append(1) + return len(s) + + + def read_fn(log: list[object]) -> object: + log.append(1) + return GFN + + + def read_type(log: list[object]) -> object: + log.append(1) + return GTYPE + + + def read_mod(log: list[object]) -> object: + log.append(1) + return GMOD + + + def call_g(x: object) -> object: + return GFN() +''') + +COMPILED = {"add2", "sub2", "mul2", "neg2", "floordiv2", "mod2", "lsh", "rsh", "roundtrip", + "compare", "fact", "grow", "consts", "next_state", "drive", "read_limit", "read_twice", + "use_builtin", "read_fn", "read_type", "read_mod", "call_g"} + +B = [0, 1, -1, 2, -2, 3, 7, -7, 2**31, 2**32, -2**31, 2**62 - 1, 2**62, 2**62 + 1, -2**62, + 2**63 - 1, 2**63 - 2, -2**63, -2**63 + 1, 3037000499, 3037000500, -3037000500] +COUNTS = [0, 1, 2, 31, 62, 63, 64, 65, 100, -1, -64] + + +def build_source(d, source, name="nimod", runtime_dir=None): + path = d / f"{name}.py" + path.write_text(source) + proved, diags = verify.verify(frontend.lower(path)) + c = cgen.generate(proved, path, path.name) + so = cbuild.build(c, path.stem, d / "out", runtime_dir=runtime_dir) + return proved, diags, c, cbuild.load(so.stem.split(".")[0], so) + + +@pytest.fixture(scope="module") +def built(tmp_path_factory): + d = tmp_path_factory.mktemp("nimod") + proved, diags, c, m = build_source(d, SOURCE) + return proved, diags, c, m + + +def deopts(m): + return m.__typedpython_deopts__ + + +def interp(m, name): + return m.__typedpython_interpreted__[name] + + +def outcome(fn, args): + log: list = [] + try: + res = ("ok", fn(log, *args)) + except BaseException as e: # noqa: BLE001 - the type and the message are observable + res = ("raise", type(e).__name__, str(e)) + return res, [(type(x).__name__, x) for x in log] + + +def differential(m, name, args): + before = deopts(m) + c = outcome(getattr(m, name), args) + after = deopts(m) + i = outcome(interp(m, name), args) + assert c == i, (name, args, c, i) + assert after == before, f"{name}{tuple(args)} deopted: the compiled path did not run" + return c + + +def fn_of(built, name): + return next(f for f in built[0].functions if f.name == name) + + +# --- red/green structure --------------------------------------------------------------------- + +def test_the_module_compiles_every_function(built): + names = {f.name for f in built[0].functions} + assert COMPILED <= names, built[0].skipped + + +def test_int_locals_of_impure_functions_are_native(built): + """i64 locals in impure functions: the local is the IR type INT, not an object.""" + for name, local in (("add2", "r"), ("fact", "f"), ("grow", "x"), ("grow", "i"), + ("roundtrip", "x"), ("compare", "x")): + f = fn_of(built, name) + assert f.pure is False + assert f.locals[local] is ir.Type.INT, (name, local, f.locals) + assert not built[1], built[1] # the verifier proved all of it + + +def test_an_int_fed_from_an_object_stays_an_object(tmp_path): + src = textwrap.dedent(''' + # typedpython: compiled + def f(log: list[object], g: object) -> object: + log.append(1) + n: int = g() + m: int = 3 + m = m + 1 + return n + ''') + proved, _, _, _ = build_source(tmp_path, src) + f = next(f for f in proved.functions if f.name == "f") + assert f.locals["n"] is ir.Type.OBJ + assert f.locals["m"] is ir.Type.INT + + +# --- the i64 boundary -------------------------------------------------------------------------- + +@pytest.mark.parametrize("name", ["add2", "sub2", "mul2", "floordiv2", "mod2"]) +def test_binary_ops_at_the_i64_boundary_equal_cpython(built, name): + m = built[3] + for a in B: + for b in B: + differential(m, name, [a, b]) + + +def test_neg_at_the_i64_boundary(built): + m = built[3] + for a in B: + differential(m, "neg2", [a]) + # the one native negation that overflows + (res, log) = differential(m, "neg2", [-2**63]) + assert res == ("ok", -2**63) and log[1:] == [("int", 2**63), ("int", -2**63)] + + +def test_shifts_at_the_i64_boundary(built): + m = built[3] + for a in B: + for n in COUNTS: + differential(m, "lsh", [a, n]) + differential(m, "rsh", [a, n]) + assert differential(m, "lsh", [1, -1])[0] == ("raise", "ValueError", "negative shift count") + assert differential(m, "lsh", [1, 63])[0] == ("ok", 2**63) + + +def test_division_by_zero_after_a_side_effect(built): + m = built[3] + for a in (0, 5, 2**63 - 1, -2**63): + (res, log) = differential(m, "floordiv2", [a, 0]) + assert res[0] == "raise" and res[1] == "ZeroDivisionError" and log, (a, res, log) + (res, log) = differential(m, "mod2", [a, 0]) + assert res[0] == "raise" and res[1] == "ZeroDivisionError" and log + + +def test_min_floordiv_minus_one_promotes(built): + m = built[3] + # -2**63 // -1 == 2**63 does not fit i64; the result is exactly the interpreter's + (res, log) = differential(m, "floordiv2", [-2**63, -1]) + assert res == ("ok", 2**63) + + +def test_promotion_survives_further_arithmetic_and_comes_back(built): + m = built[3] + for a in B: + differential(m, "roundtrip", [a]) + assert differential(m, "roundtrip", [2**63 - 1])[1][1] == ("int", 2**63 + 9) # promoted + + +def test_a_promoted_value_is_carried_through_a_loop(built): + m = built[3] + for n in (0, 1, 2, 20, 21, 25, 60): + differential(m, "fact", [n]) + assert differential(m, "fact", [25])[0] == ("ok", 15511210043330985984000000) + for a in (0, 1, -1, 2**62, 2**63 - 1, -2**63): + for k in (0, 1, 5, 45): + differential(m, "grow", [a, k]) + + +def test_comparisons_of_native_and_promoted_ints(built): + m = built[3] + for a in B: + for b in B: + differential(m, "compare", [a, b]) + + +def test_a_param_outside_i64_still_deopts_at_entry_before_any_effect(built): + m = built[3] + before = deopts(m) + log: list = [] + assert m.add2(log, 2**63, 1) == 2**63 + 2 + assert deopts(m) == before + 1 + assert log == ["start", 2**63 + 1, 2**63 + 2] + + +def test_redo_call_with_native_and_promoted_arguments(built): + m = built[3] + for seed in (0, 1, 12345, 2**40, 2**62, 2**63 - 2, 2**63 - 1, -2**63): + for n in (0, 1, 3, 40): + ca, ia = outcome(m.drive, [seed, n]), outcome(interp(m, "drive"), [seed, n]) + assert ca == ia, (seed, n, ca, ia) + + +# --- no leak, no stale reference ----------------------------------------------------------------- + +def test_promoted_values_do_not_leak(built): + m = built[3] + + def churn(): + for _ in range(300): + log: list = [] + m.fact(log, 30) + m.grow(log, 2**62, 5) + m.add2(log, 2**63 - 1, 1) + with contextlib.suppress(ZeroDivisionError): + m.floordiv2(log, 2**63 - 1, 0) + + churn() + gc.collect() + before = sys.getallocatedblocks() + churn() + gc.collect() + assert sys.getallocatedblocks() - before < 200 + + +# --- constants are built once -------------------------------------------------------------------- + +def test_constants_are_boxed_once(built): + m = built[3] + a, b = [], [] + m.consts(a) + m.consts(b) + assert a == b == [7000000000, 5, -7000000000] + assert a[0] is b[0] and a[2] is b[2], "a constant is a fresh PyLong per use, not pre-boxed" + + +def test_generated_c_boxes_no_literal(built): + c = built[2] + body = re.sub(r"/\*.*?\*/", "", c, flags=re.S) + assert not re.search(r"tp_box_i64\(\s*(?:INT64_C\()?-?\(?\d", body), \ + "a constant is boxed per use instead of read from module state" + + +# --- globals and builtins are cached behind the dict-watcher epoch ---------------------------- + +def test_global_reads_are_cached(built): + c = built[2] + assert "tp_gcache" in c + + +def test_a_rebound_global_is_seen_by_the_cached_read(built): + m = built[3] + log: list = [] + assert m.read_limit(log) == 10 + try: + for v in (11, "x", None, 2**70, 10): + m.LIMIT = v + assert differential(m, "read_limit", [])[0] == ("ok", v) + m.__dict__["LIMIT"] = 12 + assert differential(m, "read_limit", [])[0] == ("ok", 12) + m.__dict__.update(LIMIT=13) + assert differential(m, "read_limit", [])[0] == ("ok", 13) + m.__dict__.setdefault("LIMIT", 99) + assert differential(m, "read_limit", [])[0] == ("ok", 13) + del m.LIMIT + res, log = differential(m, "read_limit", []) + assert res[:2] == ("raise", "NameError") and "LIMIT" in res[2] + m.LIMIT = 10 + assert differential(m, "read_limit", [])[0] == ("ok", 10) + finally: + m.LIMIT = 10 + + +def test_a_global_rebound_in_the_middle_of_a_call(built): + m = built[3] + + def bump(): + m.LIMIT = m.LIMIT + 1 + + try: + c = outcome(m.read_twice, [bump]) + m.LIMIT = 10 + i = outcome(interp(m, "read_twice"), [bump]) + assert c == i and c[1] == [("int", 10), ("int", 11)], (c, i) + finally: + m.LIMIT = 10 + + +def test_a_patched_builtin_is_seen_by_the_cached_read(built): + m = built[3] + assert differential(m, "use_builtin", ["abc"])[0] == ("ok", 3) + real = builtins.len + try: + builtins.len = lambda s: 42 + assert differential(m, "use_builtin", ["abc"])[0] == ("ok", 42) + m.len = lambda s: 7 # a module global shadows the builtin + assert differential(m, "use_builtin", ["abc"])[0] == ("ok", 7) + del m.len + assert differential(m, "use_builtin", ["abc"])[0] == ("ok", 42) + finally: + builtins.len = real + m.__dict__.pop("len", None) + assert differential(m, "use_builtin", ["abc"])[0] == ("ok", 3) + + +# --- the cache extends no lifetime ------------------------------------------------------------ + +def _plain_function(): + def f(): + return 1 + return f + + +def _plain_type(): + return type("T", (), {}) + + +def _plain_module(): + return types.ModuleType("gm") + + +@pytest.mark.parametrize("name,reader,make", [("GFN", "read_fn", _plain_function), + ("GTYPE", "read_type", _plain_type), + ("GMOD", "read_mod", _plain_module)]) +def test_a_rebound_cached_global_is_freed_as_in_the_interpreter(built, name, reader, make): + m = built[3] + saved = getattr(m, name) + try: + for impl in (getattr(m, reader), interp(m, reader)): + old = make() + setattr(m, name, old) + for _ in range(5): # the compiled read caches it + assert impl([]) is old + ref = weakref.ref(old) + del old + setattr(m, name, None) + gc.collect() + assert ref() is None, f"{reader}: the old {name} is kept alive" + finally: + setattr(m, name, saved) + + +def test_a_finalizer_in_a_rebound_functions_dict_runs_at_the_same_point(built): + m = built[3] + saved = m.GFN + try: + traces = [] + for impl in (m.read_fn, interp(m, "read_fn")): + events: list = [] + + class D: + def __del__(self): + events.append("del") + + f = _plain_function() + f.payload = D() + m.GFN = f + for _ in range(5): + impl([]) + del f + events.append("before") + m.GFN = None + events.append("after") + traces.append(events) + assert traces[0] == traces[1] == ["before", "del", "after"], traces + finally: + m.GFN = saved + + +def test_a_global_rebound_by_one_thread_while_others_call_through_it(built): + m = built[3] + saved = m.GFN + stop = threading.Event() + bad: list = [] + + def rebinder(): + fns = [m._one, m._two] + i = 0 + while not stop.is_set(): + m.GFN = fns[i & 1] + i += 1 + + def caller(): + for _ in range(20000): + r = m.call_g(None) + if r not in (1, 2): + bad.append(r) + + try: + t = threading.Thread(target=rebinder) + cs = [threading.Thread(target=caller) for _ in range(4)] + t.start() + [c.start() for c in cs] + [c.join() for c in cs] + stop.set() + t.join() + finally: + m.GFN = saved + assert not bad, bad[:5] + + +# --- mutants: each fast path is shown to matter ------------------------------------------------- + +def mutant_runtime(tmp_path, old, new): + mutant = tmp_path / "runtime" + shutil.copytree(cbuild.RUNTIME_DIR, mutant) + text = (mutant / "tp_runtime.h").read_text() + assert old in text, old + (mutant / "tp_runtime.h").write_text(text.replace(old, new)) + return mutant + + +def test_a_wrapping_add_is_caught(tmp_path): + """Mutation: the native add does not check for overflow. The boundary test must fail.""" + rt = mutant_runtime( + tmp_path, "#define TP_INT_ADD_OVF(a, b, r) __builtin_add_overflow((a), (b), (r))", + "#define TP_INT_ADD_OVF(a, b, r) ((*(r) = (int64_t)((uint64_t)(a) + (uint64_t)(b))), 0)") + (tmp_path / "bad").mkdir() + _, _, _, bad = build_source(tmp_path / "bad", SOURCE, runtime_dir=rt) + with pytest.raises(AssertionError): + differential(bad, "add2", [2**63 - 1, 1]) + + +def test_a_cache_that_is_never_invalidated_is_caught(tmp_path): + """Mutation: the global cache ignores the epoch. A rebound global is then not seen.""" + rt = mutant_runtime( + tmp_path, "#define TP_GCACHE_FRESH(slot_epoch, epoch) ((slot_epoch) == (epoch))", + "#define TP_GCACHE_FRESH(slot_epoch, epoch) ((slot_epoch) != 0)") + (tmp_path / "bad").mkdir() + _, _, _, bad = build_source(tmp_path / "bad", SOURCE, runtime_dir=rt) + log: list = [] + assert bad.read_limit(log) == 10 + bad.LIMIT = 11 + with pytest.raises(AssertionError): + differential(bad, "read_limit", []) + + +# --- the cost rule and wordfreq -------------------------------------------------------------- + +def test_an_int_loop_is_native_work_for_the_cost_rule(tmp_path): + src = textwrap.dedent(''' + # typedpython: compiled + def digits(log: list[object], n: int) -> None: + x: int = n + 1 + while x > 0: + log.append(x % 10) + x //= 10 + ''') + path = tmp_path / "digits.py" + path.write_text(src) + proved, diags = verify.verify(frontend.lower(path)) + kept, diags = cost.apply(proved, diags) + assert {f.name for f in kept.functions} == {"digits"}, kept.skipped + + +def test_an_object_only_loop_is_still_rejected(tmp_path): + src = textwrap.dedent(''' + # typedpython: compiled + def words(xs: list[object], ys: list[object]) -> None: + for x in xs: + ys.append(x) + ''') + path = tmp_path / "words.py" + path.write_text(src) + proved, diags = verify.verify(frontend.lower(path)) + kept, diags = cost.apply(proved, diags) + assert {f.name for f in kept.functions} == set(), kept.skipped + assert "policy/cost" in kept.skipped["words"] + + +@pytest.fixture(scope="module") +def wordfreq(tmp_path_factory): + d = tmp_path_factory.mktemp("wordfreq") + src = "# typedpython: compiled\n" + WORDFREQ.read_text() + path = d / "wordfreq.py" + path.write_text(src) + proved, diags = verify.verify(frontend.lower(path)) + kept, diags = cost.apply(proved, diags) + c = cgen.generate(kept, path, path.name) + so = cbuild.build(c, path.stem, d / "out") + m = cbuild.load(so.stem.split(".")[0], so) + spec = importlib.util.spec_from_file_location("wordfreq_plain", WORDFREQ) + plain = importlib.util.module_from_spec(spec) + spec.loader.exec_module(plain) + return kept, diags, m, plain + + +def run_main(main, n): + old = sys.argv + sys.argv = ["wordfreq", str(n)] + buf = io.StringIO() + try: + with contextlib.redirect_stdout(buf): + main() + finally: + sys.argv = old + return buf.getvalue() + + +def test_wordfreq_word_of_and_main_pass_the_cost_rule(wordfreq): + kept, diags, _, _ = wordfreq + names = {f.name for f in kept.functions} + assert {"next_state", "word_of", "main"} <= names, kept.skipped + assert not [d for d in diags if d.rule == "policy/cost"] + + +def test_wordfreq_output_equals_cpython(wordfreq): + _, _, m, plain = wordfreq + for n in (0, 1, 2, 50, 3000): + assert run_main(m.main, n) == run_main(plain.main, n) + assert run_main(m.main, n) == run_main(interp(m, "main"), n) + assert deopts(m) == 0 + for i in (0, 1, 25, 26, 17575, 17576, 99999, 10**9): + assert m.word_of(i) == plain.word_of(i) diff --git a/python-multiplatform-ksp/src/test/python/test_runtime.py b/python-multiplatform-ksp/src/test/python/test_runtime.py index 629798e7..18d1b22d 100644 --- a/python-multiplatform-ksp/src/test/python/test_runtime.py +++ b/python-multiplatform-ksp/src/test/python/test_runtime.py @@ -1203,4 +1203,4 @@ def test_binop_obj_maps_the_six_kinds_to_the_python_operators(rt): assert rt.binop_obj(I64_MAX, 1, 0) == 2**63 assert rt.binop_obj(I64_MIN, -1, 4) == 2**63 assert exc_of(rt.binop_obj, 1, 0, 4)[1:3] == (ZeroDivisionError, str(exc_of(operator.floordiv, 1, 0)[2])) - assert exc_of(rt.binop_obj, 1, 2, 12)[1] is SystemError # 6..11 are the in-place forms (#150) + assert exc_of(rt.binop_obj, 1, 2, 16)[1] is SystemError # 6..11 in-place (#150), 12..15 shifts (#232)