Skip to content

vm: cut per-call and per-iteration overhead in the dispatcher - #401

Merged
davydog187 merged 4 commits into
mainfrom
perf/vm-call-path
Jul 27, 2026
Merged

davydog187 merged 4 commits into
mainfrom
perf/vm-call-path

Conversation

@davydog187

@davydog187 davydog187 commented Jul 27, 2026 •

Copy link
Copy Markdown
Contributor

Cuts per-call and per-iteration overhead in the bytecode dispatcher. Runtime only — the bytecode format, the encoder, and codegen are untouched, so the codegen work (peephole, static-arity calls, new opcodes) lands independently on top of this.

Two commits, each independently green on the full suite.

1. vm: inline hot-path guards and fast paths in the dispatcher

Mechanical micro-fixes. Nothing changes semantics; each removes work the default configuration paid for a result it already knew.

  • close_upv/2, tick/2, ckdepth/1 — inlined local shims so the empty-open_upvalues, max_instructions: :infinity and max_call_depth: :infinity cases resolve in one function-head match instead of a cross-module call into Executor/State. The empty-open-upvalues guard alone was two cross-module calls per loop iteration.
  • Call frames build {source, line, name_hint} inline instead of through Executor.dispatcher_call_info/3.
  • setup_vararg_proto/4 is guarded by the callee's is_vararg at the call site, so a non-vararg call pays one field read. (The vararg path still copies the whole %Prototype{}.)
  • Map.get/2 → :maps.get/3 on the upvalue-cell and open-upvalue reads, killing the Map.get/2 → Map.get/3 hop.
  • @op_set_field gets the inline no-metatable fast path @op_get_field already had. Tables carrying a metatable still bridge, so the __newindex chain stays exactly the interpreter's.
  • Table.from_data/1 short-circuits the empty map — every {} constructor was paying Enum.sort plus a fold over nothing.
  • Table.insert_hash/3 gains a no-dead-keys clause, skipping the resurrection probe on tables that have never had a key removed.
  • The compiled-mode native-call position bridge (on the path of every stdlib call) uses the process-dictionary BIFs directly and restores explicitly rather than through a try/after. Executor.execute/5's after restore_position/1 remains the outer net.

2. vm: thread call stack, depth, and open upvalues as loop parameters

Every in-mode Lua call rebuilt the 17-field %State{} twice — once to push call_stack/call_depth and reset open_upvalues, once to undo all three — for ~42 words of garbage per call on a path that otherwise allocates almost nothing.

These are control state, not heap state, so they now follow the discipline instruction_count already used: parameters of the dispatch loop, living in X-registers, allocating nothing. Pushing a frame is a cons plus an increment; popping is tl/1 plus a decrement, with the caller's open upvalues coming back off the frame tuple that already carried them.

%State{} keeps the fields — the interpreter still stores them there, and everything outside the loop reads them from the struct (debug.getinfo, error(msg, level), traceback formatting, State.check_call_depth!/1, State.tick!/2). So the loop writes them back (sync/3) wherever execution leaves it: an Executor bridge, a native callback, its own raise sites. sync/3 returns the struct untouched when it already agrees, so a loop body that bridges every iteration without calling anything (t[k] = v, an __index read) pays the write once instead of per iteration.

This differs from the sketch in the brief, which proposed dropping state.call_stack entirely and materializing the traceback lazily from the dispatcher's frames list on the error path. Threading it eagerly was chosen instead: the frame entry is 4 words plus a cons against the 42 the struct rebuild cost, so nearly all of the win is kept, while lazy materialization would rebuild an O(depth) list at every bridge crossing — a regression for bridge-heavy code at depth (OOP self.x on a table with a metatable bridges on every read) — and would have needed the traceback reconstructed from call opcodes rather than pushed by them.

close_upv/2 now filters the parameter directly, retiring Executor.dispatcher_close_open_upvalues_at_or_above/2; Executor.dispatcher_call_info/3 went unused in commit 1 and goes with it, its frame-shape rationale moving to the call_frame/0 typedoc.

Debug fidelity

Unchanged by construction — the same {source, line, name_hint} frames in the same order, pushed onto a parameter rather than a struct field. Verified by diffing Lua.format_exception/1 output across thirteen failing programs (nested runtime errors, error() at levels 1 and 2, stdlib bad-argument raises, __index arithmetic errors, debug.traceback from depth, debug.getinfo currentline/name, error-after-caught-error, concat errors, deep recursion tracebacks) plus the exact trip points of :max_instructions across 7 budgets and :max_call_depth across 11 depths. Byte-identical before and after, including the 200-frame ... 190 more frames ... traceback elision and pcall-caught "stack overflow".

Numbers

MIX_ENV=prod, min-of-9 × 100 reps per measurement, alternating A/B by swapping the source files in place and recompiling between runs. The machine had concurrent load: within-configuration spread across rounds was up to 5%, so these are the best round for each configuration, and the alternation is what makes the comparison meaningful rather than any single number.

Workload bodies are wrapped in a Lua function, because Lua.VM.execute/3 runs the main chunk on the interpreter and only enters the dispatcher through a call.

Per commit:

workload main after commit 1 after commit 2
fib(15) 479.9 µs 459.2 µs (-4.3%) 365.5 µs (-22.7% on commit 1)
closures 100×10 506.7 µs 493.2 µs (-2.7%) 448.5 µs (-11.0%)
oop 2000 4590.1 µs 4468.6 µs (-2.6%) 4386.8 µs (-4.6%)
table build 5000 3799.3 µs 3633.3 µs (-4.4%) 3792.7 µs (-0.7%)
set_field 20000 5961.0 µs 5588.4 µs (-6.3%) 5659.4 µs (+0.5%)

End to end, main vs the branch tip, measured in one alternating run:

workload main branch
fib(15) 488.2 µs 363.8 µs 1.34×
closures 100×10 527.7 µs 438.4 µs 1.20×
oop 2000 4660.2 µs 4306.4 µs 1.08×
table build 5000 3833.9 µs 3672.6 µs 1.04×
set_field 20000 5979.5 µs 5527.7 µs 1.08×

3. vm: pin dispatcher-side open-upvalue isolation across require

One regression test. The existing require open-upvalue tests only exercise a main chunk, which runs on the interpreter; the invariant they pin (a nested evaluation cannot see the caller's register-keyed upvalue cells) now holds on the dispatcher side by construction rather than by a save/restore, so it gets a test entered from compiled code.

Validation

  • mix format --check-formatted clean; no compiler warnings.
  • mix test --include slow --include lua53 — 2639 passed, 16 skipped (includes the official Lua 5.3 suite and the error-message/traceback tests) after each commit.
  • mix test --include differential — 2618 passed.
  • mix docs --warnings-as-errors clean; mix dialyzer clean (0 errors).

Mechanical micro-fixes on the per-call and per-iteration path. None of
them changes semantics; each removes work the default configuration was
paying for a result it already knew.

  * `close_upv/2`, `tick/2`, `ckdepth/1`: inlined local shims so the
    empty-`open_upvalues`, `max_instructions: :infinity` and
    `max_call_depth: :infinity` cases resolve in one function-head match
    instead of a cross-module call into `Executor`/`State`.
  * Call frames build the `{source, line, name_hint}` tuple inline
    rather than through `Executor.dispatcher_call_info/3`.
  * `setup_vararg_proto/4` is guarded by the callee's `is_vararg` at the
    call site, so a non-vararg call pays one field read.
  * `Map.get/2` -> `:maps.get/3` on the upvalue-cell and open-upvalue
    reads, dropping the `Map.get/2` -> `Map.get/3` hop.
  * `@op_set_field` gets the inline no-metatable fast path that
    `@op_get_field` already had; tables carrying a metatable still
    bridge so the `__newindex` chain stays the interpreter's.
  * `Table.from_data/1` short-circuits the empty map, so a `{}`
    constructor no longer pays `Enum.sort` plus a fold.
  * `Table.insert_hash/3` gains a no-dead-keys clause, skipping the
    resurrection probe on tables that have never had a key removed.
  * The compiled-mode native-call position bridge uses the process
    dictionary BIFs directly and restores explicitly instead of through
    a `try/after`; `Executor.execute/5`'s `after` remains the outer net.

Measured (MIX_ENV=prod, min-of-9 x 100 reps, alternating A/B over four
rounds on a machine with concurrent load; best round per configuration):

    fib(15)              479.9us -> 459.2us   -4.3%
    closures 100x10      506.7us -> 493.2us   -2.7%
    oop 2000            4590.1us -> 4468.6us  -2.6%
    table build 5000    3799.3us -> 3633.3us  -4.4%
    set_field 20000     5961.0us -> 5588.4us  -6.3%
Every in-mode Lua call used to rebuild the 17-field `%State{}` twice —
once to push `call_stack`/`call_depth` and reset `open_upvalues`, once to
undo all three on the way out — for roughly 42 words of garbage per call
on a path that otherwise allocates almost nothing.

These three fields are control state, not heap state, so they follow the
discipline `instruction_count` already used: they become parameters of
the dispatch loop, living in X-registers and allocating nothing. Pushing
a frame is now a cons onto the parameter plus an increment; popping it is
`tl/1` plus a decrement, with the caller's open upvalues coming back off
the frame tuple that already carried them.

`%State{}` keeps the fields, because the interpreter still stores them
there and because everything outside the loop reads them from the struct:
`debug.getinfo`, `error(msg, level)`, traceback formatting,
`State.check_call_depth!/1`, `State.tick!/2`. The loop therefore writes
them back — `sync/3` — wherever execution leaves it: an `Executor`
bridge, a native callback, its own raise sites. `sync/3` returns the
struct untouched when it already agrees, so a loop body that bridges
every iteration without calling anything (`t[k] = v`, an `__index` read)
pays for the write once rather than per iteration.

`close_upv/2` now filters the parameter directly, which retires
`Executor.dispatcher_close_open_upvalues_at_or_above/2`;
`Executor.dispatcher_call_info/3` went unused in the previous commit and
goes with it, its frame-shape rationale moving to the `call_frame/0`
typedoc.

Debug output is unchanged by construction: the same
`{source, line, name_hint}` frames are pushed in the same order, just
onto a parameter. Verified by diffing `Lua.format_exception/1` output
across thirteen failing programs (nested runtime errors, `error()` at
levels 1 and 2, stdlib bad-argument raises, `__index` arithmetic errors,
`debug.traceback`, `debug.getinfo`, error-after-caught-error) plus the
exact trip points of `:max_instructions` (7 budgets) and
`:max_call_depth` (11 depths) — byte-identical before and after.

Measured (MIX_ENV=prod, min-of-9 x 100 reps, alternating A/B over three
rounds on a machine with concurrent load; best round per configuration):

    fib(15)              472.8us -> 365.5us   -22.7%  (1.29x)
    closures 100x10      504.0us -> 448.5us   -11.0%
    oop 2000            4598.3us -> 4386.8us   -4.6%
    table build 5000    3819.1us -> 3792.7us   -0.7%
    set_field 20000     5633.8us -> 5659.4us   +0.5%
The dispatcher now carries its open upvalues as a loop parameter rather
than in `%State{}`, so a nested evaluation is isolated from them by
construction instead of by a save/restore. Pin that from the compiled
side: the interpreter-entered variants above only exercise a main chunk.
@davydog187

Copy link
Copy Markdown
Contributor Author

🤖 Automated review — Opus + Sonnet consolidated findings

Two independent AI reviewers (Opus, Sonnet) reviewed this PR. Both converged on the same MEDIUM. The cs/cd/ou threading itself was verified clean by both: tick/depth shims are clause-identical to State.tick!/check_call_depth! (including :infinity no-increment), pushes/pops are balanced across all 20 bridge sites, open-upvalue isolation holds, and unwind_to/2 keeps pcall recovery intact.

🟠 MEDIUM — position-restore is no longer exception-safe in the compiled-mode native-call bridge (found independently by both reviewers)

lib/lua/vm/executor.ex:583-591 (dispatcher_call_function/6, {:native_func, _} clause)

The PR replaces the try/after restore_position(prev_pos) with a sequential restore that only runs on normal return. A raise inside the wrapped native call leaves the stale {line, source} in @position_key, and the compiled path has no outer safety net: Dispatcher.do_execute_top/4 is try/rescue with no after, and Lua.call_function/3 → do_call_function is rescue-only.

Concrete scenario: Lua.call_function/3 on a compiled closure raises inside a stdlib call (say string.rep) at chunk-a:1 → position leaks; a later Lua.eval! of chunk-b hits a line: nil bridge error (e.g. index a nil) → Lua.format_exception/1 renders at chunk-b:1 with a wrongly-highlighted source-context line, where main rendered no line. Lua-visible pcall values are unaffected (host diagnostics only). The pcall/xpcall paths self-repair because those handlers return normally through this same bridge — which is why the PR's manual "error-after-caught-error" checks came out byte-identical; the unprotected path is no-pcall / host-boundary.

The two structurally identical interpreter-side native-call bridges in the same file ({:native_func, fun} clause of do_execute ~line 1465, and call_value/5 ~line 2880) still use try/after — so this reads as an oversight, not a design choice.

🟡 LOW

  • Same site: the xpcall message handler now runs with the failing call's position published instead of the xpcall site's (the old after clauses ran during unwinding). Arguably more accurate, but an untested behavior change.
  • lib/lua/vm/executor.ex:188-192 — comment claims the dispatcher does its own open-upvalue "save/restore"; it now isolates via a loop parameter. Stale.
  • Coverage: the new inline @op_set_field fast path has no compiled-code test for the metatable→bridge arm or the t.x = nil delete arm (verified equivalent by inspection); test/lua/vm/require_open_upvalue_test.exs:92-124 doesn't assert the subject function was actually bytecode-compiled (first_sub(proto).bytecode), so it can silently degrade into an interpreter test.

Follow-up (pre-existing, out of scope)

sync/3 doesn't stamp instruction_count, so metamethod work behind a bridge is charged against a stale seed and discarded — :max_instructions under-counts metamethod-heavy code. Unchanged from main; will be filed as an issue given the sandboxing focus.

A fix for the MEDIUM and the LOW items will be pushed to this branch shortly.

The dispatcher's native-call bridge published {line, source} into the
position slot and restored it with a sequential `case` that only runs on
a normal return. A raise inside the wrapped stdlib call skipped the
restore, and not every entry into this bridge sits under `execute/5`'s
`after restore_position/1` net: `Lua.call_function/3` on a compiled
closure is rescue-only, as is `Dispatcher.do_execute_top/4`. The stale
position then survived the call and attributed a later, unrelated
evaluation's line-less bridge error (indexing nil inside a compiled
body) to the first chunk's line.

Restores `try/after` around the call, keeping the BIF-level process
dictionary access the sibling interpreter bridges' shape allows.

Also:
  * correct the `:compiled_closure` comment — the dispatcher isolates
    open upvalues via a dispatch-loop parameter, not save/restore
  * cover the inlined `:set_field` arms from compiled code: `__newindex`
    with both function and table handlers, and the nil-delete case
  * assert the require/open-upvalue dispatcher test's subject actually
    bytecode-compiles so it can't degrade into an interpreter test

Claude-Session: https://claude.ai/code/session_01BYHGFLoHBJAsUrTzjVp5nC
@davydog187

Copy link
Copy Markdown
Contributor Author

Pushed 694a579 addressing the review findings. The MEDIUM one is real: the dispatcher's native-call bridge lost its try/after, so a raise inside a compiled-mode stdlib call left {line, source} in the position slot — and Lua.call_function/3 on a compiled closure is rescue-only, with no outer after restore_position/1 net, so the stale line then attributed a later unrelated evaluation's line-less bridge error to the first chunk. try/after is back (BIF-level pdict access kept), with a regression test in test/lua/call_function_error_value_test.exs that reproduces the cross-evaluation leak — verified failing before the fix (line: 4 from chunk one on a second.lua error) and passing after. Also corrected the stale :compiled_closure comment to describe the dispatch-loop-parameter isolation, added compiled-code coverage for the inlined :set_field arms (__newindex function + table handlers, nil-delete), and pinned the require/open-upvalue test's subject as actually bytecode-compiled. Full suite green: 2623 passed.

@davydog187
davydog187 merged commit 983da9d into main Jul 27, 2026
5 checks passed
@davydog187
davydog187 deleted the perf/vm-call-path branch July 27, 2026 19:19
@davydog187 davydog187 mentioned this pull request Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant